Garbage collection in JVM and specifics of huge heaps Alexey Ragozin [email_address] Oct 2011
OVERVIEW OF  GARBAGE COLLECTION
Garbage collection <ul><li>Stop-the-world (STW) pause  – pause of all application threads require  </li></ul><ul><li>Compa...
Garbage collection <ul><li>Copy collection </li></ul><ul><ul><li>Traverse object graph and copy reachable object to other ...
Garbage collection <ul><li>S  – whole heap size </li></ul><ul><li>L  – size of live objects </li></ul><ul><li>Copy collect...
Garbage collection Generational approach
Garbage collection <ul><li>Young space collection </li></ul><ul><ul><li>High throughput </li></ul></ul><ul><ul><li>Low mem...
Oracle’s HotSpot JVM <ul><li>Default (serial) collector </li></ul><ul><li>Young:  Serial copy collector,  Old:  serial MSC...
Oracle’s HotSpot JVM
Oracle’s JRockit JVM <ul><li>Pick any flavor of mark/sweep/compact </li></ul><ul><li>Generational or single space </li></u...
Oracle’s JRockit JVM http:// aragozin.blogspot.com/2011/07/jrockit-gc-in-action.html -Xgc:  option Generational Mark Sweep...
IBM’s J9 <ul><li>-Xgcpolicy:optthruput  </li></ul><ul><li>Single space, stop-the-world collector </li></ul><ul><li>-Xgcpol...
Azul’s Zing JVM <ul><li>Generational collection </li></ul><ul><li>Young space – concurrent MSC </li></ul><ul><li>Old space...
YOUNG COLLECTION
Tuning CMS for low pauses <ul><li>Memory geometry </li></ul><ul><li>Young space:  -XX:NewSize=<n> -XX:MaxNewSize=<n> </li>...
How young collection works? <ul><li>Collect root references </li></ul><ul><ul><li>Stack frame references </li></ul></ul><u...
How young collector works Collecting root references
How young collection works? Coping live objects
How young collection works? Collection finished
Card marking write barrier
Thread local allocation blocks <ul><li>Each thread preallocates block in Eden </li></ul><ul><li>Thread is allocating new o...
Young collection stop-the-world <ul><li>Total STW time </li></ul><ul><ul><li>Collect roots </li></ul></ul><ul><ul><ul><li>...
Tuning young collection <ul><li>Eden size </li></ul><ul><li>too small  – frequent YGC, objects promoted to old space early...
Tuning young collection <ul><li>Eden size </li></ul><ul><li>-XX:MaxNewSize=<n> -XX:NewSize=<n> </li></ul><ul><li>Eden size...
Tuning young collection <ul><li>Small heap sizes </li></ul><ul><li>Balance tenuring threshold / survivor space to keep obj...
Tuning young collection Research mode
Tuning young collection <ul><li>GC tuning is based on application allocation pattern </li></ul><ul><li>If application allo...
OLD SPACE COLLECTION
Old space collection <ul><li>HotSpot </li></ul><ul><li>Parallel </li></ul><ul><li>Concurrent Mark Sweep (CMS) </li></ul><u...
Old space collection <ul><li>HotSpot’s G1 </li></ul><ul><li>Space is divided into regions </li></ul><ul><li>Regions can be...
Old space collection <ul><li>HotSpot’s CMS (Concurrent Mark Sweep) </li></ul><ul><li>Does not compact </li></ul><ul><li>Pr...
Tuning CMS for low pauses <ul><li>Young collection STW pause </li></ul><ul><li>Initial marking STW pause </li></ul><ul><li...
Cost structure of pauses (CMS) Summary of pauses
Patching OpenJDK Vicious loop http://aragozin.blogspot.com/2011/07/openjdk-patch-cutting-down-gc-pause.html void  ClearNon...
Patching OpenJDK Vicious loop - Optimized void  ClearNoncleanCardWrapper :: do_MemRegion ( MemRegion mr )   { ... while   ...
Patching OpenJDK Serial collector gain http://aragozin.blogspot.com/2011/07/openjdk-patch-cutting-down-gc-pause.html
Patching OpenJDK CMS collector gain http://aragozin.blogspot.com/2011/07/openjdk-patch-cutting-down-gc-pause.html
CMS, PLABs and fragmentation
CMS, PLABs and fragmentation
CMS, PLABs and fragmentation
CMS, PLABs and fragmentation
JRockit’s gencon Hidden pauses
Tuning old space <ul><li>-Xms<N> -Xmx<N> </li></ul><ul><li>Young space is not for object, it is for garbage </li></ul><ul>...
OTHER ASPECTS OF GC IN JAVA
JNI and pinning objects <ul><li>Normally JNI code is working with copy of java object. </li></ul><ul><li>Native code my re...
Finalizers <ul><li>JVM have to keep special reference for object eligible for finalization </li></ul><ul><li>Object with f...
Soft references <ul><li>In server mode, JVM cleans soft reference if it was not dereferenced during N milliseconds. N is p...
SURVIVING HUGE HEAP CONCLUSION
GC tuning summary <ul><li>Young space tuning </li></ul><ul><li>Find balance between, pause frequency, pause time (time to ...
Surviving with huge heap <ul><li>CMS is very good in terms of pauses </li></ul><ul><ul><li>You can reliably keep pauses un...
MOVING OUT OF HEAP
Direct memory buffers <ul><li>java.nio.ByteBuffer.allocateDirect() </li></ul><ul><li>Pro </li></ul><ul><li>Memory is alloc...
RTSJ <ul><li>Scoped memory </li></ul><ul><li>Objects can be allocated in chosen memory areas </li></ul><ul><li>Scoped and ...
Unsafe java <ul><li>sun.misc.Unsafe </li></ul><ul><li>Unsafe.allocateMemory(…) </li></ul><ul><li>Unsafe.reallocateMemory(…...
<ul><li>Thank you </li></ul>Alexey Ragozin  [email_address] http://aragozin.blogspot.com - my articles
Prochain SlideShare
Chargement dans... 5
×

Garbage collection in JVM

8,150

Published on

Practical overview of garbage collection in different JVMs with emphasis on tuning, large heaps and minimizing pause times
UPDATED 13 Oct 2011

Published in: Technologies, Éducation
2 commentaires
32 mentions J'aime
Statistiques
Remarques
  • Hi
    i am new to GC
    can some help me on how to calculate application stopped time for the following JVM's
    1.JRocket
    2.IBM
    i did this for SUN by using -XX:+PrintGCApplicationStoppedTime
    but i didn't found any flag for above two JVM's
    or
    Any other way to calculate application stopped time
    Plz explain me
       Répondre 
    Êtes-vous sûr de vouloir  Oui  Non
    Votre message apparaîtra ici
  • Hi
    i am new to GC
    can some help me on how to calculate application stopped time for the following JVM's
    1.JRocket
    2.IBM
    i did this for SUN by using -XX:+PrintGCApplicationStoppedTime
    but i didn't found any flag for above two JVM's
    or
    Any other way to calculate application stopped time
    Plz explain me
       Répondre 
    Êtes-vous sûr de vouloir  Oui  Non
    Votre message apparaîtra ici
Aucun téléchargement
Vues
Total des vues
8,150
Sur Slideshare
0
À partir des ajouts
0
Nombre d'ajouts
3
Actions
Partages
0
Téléchargements
0
Commentaires
2
J'aime
32
Ajouts 0
No embeds

No notes for slide

Garbage collection in JVM

  1. 1. Garbage collection in JVM and specifics of huge heaps Alexey Ragozin [email_address] Oct 2011
  2. 2. OVERVIEW OF GARBAGE COLLECTION
  3. 3. Garbage collection <ul><li>Stop-the-world (STW) pause – pause of all application threads require </li></ul><ul><li>Compacting algorithms – can move objects in memory to defragment free space </li></ul><ul><li>Parallel collection – using multiple cores to reduce STW time </li></ul><ul><li>Concurrent collection – collection algorithms working in parallel with application threads </li></ul>Terms dictionary
  4. 4. Garbage collection <ul><li>Copy collection </li></ul><ul><ul><li>Traverse object graph and copy reachable object to other space </li></ul></ul><ul><ul><li>Mark old space as free </li></ul></ul><ul><li>Mark / Sweep </li></ul><ul><ul><li>Traverse object graph and mark reachable objects </li></ul></ul><ul><ul><li>Scan (sweep) whole memory and “free” unmarked objects </li></ul></ul><ul><li>Mark / Sweep / Compact </li></ul><ul><ul><li>… mark … sweep …. </li></ul></ul><ul><ul><li>Relocate live objects to defragment free space </li></ul></ul>Algorithms
  5. 5. Garbage collection <ul><li>S – whole heap size </li></ul><ul><li>L – size of live objects </li></ul><ul><li>Copy collection </li></ul><ul><li>Throughput </li></ul><ul><li>Mark / Sweep </li></ul><ul><li>Throughput </li></ul><ul><li>For all algorithms based on reference reachability. GC efficiency is in reverse proportion to amount of live objects . </li></ul>Economics
  6. 6. Garbage collection Generational approach
  7. 7. Garbage collection <ul><li>Young space collection </li></ul><ul><ul><li>High throughput </li></ul></ul><ul><ul><li>Low memory utilization </li></ul></ul><ul><li>Promotion </li></ul><ul><ul><li>Eden (nursery) -> Survivor (keep) space -> Old space </li></ul></ul><ul><li>Old space collection </li></ul><ul><ul><li>Better memory utilization </li></ul></ul><ul><ul><li>Orders of magnitude lower throughput </li></ul></ul><ul><li>Memory barrier </li></ul><ul><ul><ul><li>JVM “tracks” references from old to young space </li></ul></ul></ul>Generational approach
  8. 8. Oracle’s HotSpot JVM <ul><li>Default (serial) collector </li></ul><ul><li>Young: Serial copy collector, Old: serial MSC </li></ul><ul><li>Parallel scavenge / Parallel old GC </li></ul><ul><li>Young: Parallel copy collector, Old: serial MSC or parallel MSC </li></ul><ul><li>Concurrent mark sweep (CMS) </li></ul><ul><li>Young: Serial or parallel copy collector, Old: concurrent mark sweep </li></ul><ul><li>G1 (garbage first) </li></ul><ul><li>Young: Copy collector (region based) Old: Incremental MSC </li></ul>http://aragozin.blogspot.com/2011/07/hotspot-jvm-garbage-collection-options.html
  9. 9. Oracle’s HotSpot JVM
  10. 10. Oracle’s JRockit JVM <ul><li>Pick any flavor of mark/sweep/compact </li></ul><ul><li>Generational or single space </li></ul><ul><li>Young collection – parallel copy collector </li></ul><ul><li>Marking phase – parallel STW or concurrent </li></ul><ul><li>Compact phase – parallel STW or incremental (concurrent) </li></ul>http:// aragozin.blogspot.com/2011/07/jrockit-gc-in-action.html
  11. 11. Oracle’s JRockit JVM http:// aragozin.blogspot.com/2011/07/jrockit-gc-in-action.html -Xgc: option Generational Mark Sweep/Compact genconcon or gencon Yes concurrent incremental singleconcon or singlecon No concurrent incremental genconpar Yes concurrent parallel singleconpar No concurrent parallel genparpar or genpar Yes parallel parallel singleparpar or singlepar No parallel parallel genparcon Yes parallel incremental singleparcon No parallel incremental
  12. 12. IBM’s J9 <ul><li>-Xgcpolicy:optthruput </li></ul><ul><li>Single space, stop-the-world collector </li></ul><ul><li>-Xgcpolicy:optavgpause </li></ul><ul><li>Single space, partially concurrent collector </li></ul><ul><li>-Xgcpolicy:gencon </li></ul><ul><li>Generational, partially concurrent collector </li></ul>
  13. 13. Azul’s Zing JVM <ul><li>Generational collection </li></ul><ul><li>Young space – concurrent MSC </li></ul><ul><li>Old space – concurrent MSC </li></ul><ul><li>Unlike competitors, Azul can do compaction without STW pause. </li></ul><ul><li>Secret is using of read barrier. </li></ul>
  14. 14. YOUNG COLLECTION
  15. 15. Tuning CMS for low pauses <ul><li>Memory geometry </li></ul><ul><li>Young space: -XX:NewSize=<n> -XX:MaxNewSize=<n> </li></ul><ul><li>Survival space: Young space / -XX:SurvivorRatio=<n> </li></ul><ul><li>Young + old space: -Xms<n> -Xmx<n> </li></ul><ul><li>Permanent space: -XX:PermSize=<n> -XX:MaxPerSize=<n> </li></ul>Memory in HotSpot JVM
  16. 16. How young collection works? <ul><li>Collect root references </li></ul><ul><ul><li>Stack frame references </li></ul></ul><ul><ul><li>References from other spaces (tenured + permanent) </li></ul></ul><ul><li>Travers object graph </li></ul><ul><ul><li>Visit only live object </li></ul></ul><ul><ul><li>Copy live object to other region of young space or old space </li></ul></ul><ul><li>Consider whole Eden and old survivor space to be free memory </li></ul><ul><li>Write barrier is required to effectively collect references from old to young space. </li></ul>
  17. 17. How young collector works Collecting root references
  18. 18. How young collection works? Coping live objects
  19. 19. How young collection works? Collection finished
  20. 20. Card marking write barrier
  21. 21. Thread local allocation blocks <ul><li>Each thread preallocates block in Eden </li></ul><ul><li>Thread is allocating new objects in its TLAB </li></ul><ul><li>Then TLAB is full, new TLAB allocated </li></ul><ul><li>If object does not fit TLAB </li></ul><ul><ul><li>Allocate in Eden space </li></ul></ul><ul><ul><ul><li>If does not fit Eden (or ‑XX:PretenureSizeThreshold ) </li></ul></ul></ul><ul><ul><ul><ul><li>Allocate in old space </li></ul></ul></ul></ul>
  22. 22. Young collection stop-the-world <ul><li>Total STW time </li></ul><ul><ul><li>Collect roots </li></ul></ul><ul><ul><ul><li>Scan thread stacks </li></ul></ul></ul><ul><ul><ul><li>Scan dirty cards </li></ul></ul></ul><ul><ul><ul><ul><li>Read card table ~ S heap </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Scan pages marked as dirty ~ </li></ul></ul></ul></ul><ul><ul><li>Copy live objects </li></ul></ul><ul><ul><li>Process special references </li></ul></ul>* You can use -XX:+PrintGCTaskTimeStamps to analyze time of individual phases * You can use -XX:+PrintReferenceGC to analyze reference processing times
  23. 23. Tuning young collection <ul><li>Eden size </li></ul><ul><li>too small – frequent YGC, objects promoted to old space early </li></ul><ul><li>too large – more long lived objects need to be copied </li></ul><ul><li>Survivor space size </li></ul><ul><li>too small – overflow, objects prematurely promoted </li></ul><ul><li>too large – memory wasted </li></ul><ul><li>Tenuring threshold </li></ul><ul><li>higher – objects are kept in young space for longer </li></ul><ul><li>higher – more objects in young space, more copy time </li></ul>
  24. 24. Tuning young collection <ul><li>Eden size </li></ul><ul><li>-XX:MaxNewSize=<n> -XX:NewSize=<n> </li></ul><ul><li>Eden size = new size – 2 * survivor space size </li></ul><ul><li>Survivor space size </li></ul><ul><li>-XX:SurvivorRatio=<n> </li></ul><ul><li>Survivor space size = new size / survivor ratio </li></ul><ul><li>Tenuring threshold </li></ul><ul><li>-XX:MaxTenuringThreshold=<n> </li></ul>
  25. 25. Tuning young collection <ul><li>Small heap sizes </li></ul><ul><li>Balance tenuring threshold / survivor space to keep objects in limited young space for longer </li></ul><ul><li>Large heap sizes (4Gb and greater) </li></ul><ul><li>Limit tenuring threshold to avoid increase in copy time </li></ul><ul><li>Limit survivor space to avoid accidental long young collections </li></ul><ul><li>Increase Eden size instead of increasing tenuring threshold </li></ul>
  26. 26. Tuning young collection Research mode
  27. 27. Tuning young collection <ul><li>GC tuning is based on application allocation pattern </li></ul><ul><li>If application allocation patterns is changed – you are in trouble </li></ul><ul><li>In practice application always have different “modes of operation” </li></ul><ul><li>GC tuning – choosing better evil </li></ul>
  28. 28. OLD SPACE COLLECTION
  29. 29. Old space collection <ul><li>HotSpot </li></ul><ul><li>Parallel </li></ul><ul><li>Concurrent Mark Sweep (CMS) </li></ul><ul><li>G1 </li></ul><ul><li>JRockit </li></ul><ul><li>Parallel </li></ul><ul><li>Incremental (concurrent) </li></ul>
  30. 30. Old space collection <ul><li>HotSpot’s G1 </li></ul><ul><li>Space is divided into regions </li></ul><ul><li>Regions can be collected individually </li></ul><ul><li>Write barrier between regions </li></ul><ul><li>JRockit’s concurrent (increamental) </li></ul><ul><li>Space is divided into regions </li></ul><ul><li>Mark/Sweep whole heap, compact just a fraction </li></ul>
  31. 31. Old space collection <ul><li>HotSpot’s CMS (Concurrent Mark Sweep) </li></ul><ul><li>Does not compact </li></ul><ul><li>Prone to fragmentation </li></ul><ul><li>Use separate free lists for each object size </li></ul><ul><li>Use statistic to manage fragmentation </li></ul>
  32. 32. Tuning CMS for low pauses <ul><li>Young collection STW pause </li></ul><ul><li>Initial marking STW pause </li></ul><ul><li>Remark STW pause </li></ul><ul><li>Promotion failure / Full GC pause </li></ul>Summary of pauses
  33. 33. Cost structure of pauses (CMS) Summary of pauses
  34. 34. Patching OpenJDK Vicious loop http://aragozin.blogspot.com/2011/07/openjdk-patch-cutting-down-gc-pause.html void ClearNoncleanCardWrapper :: do_MemRegion ( MemRegion mr ) { … while ( cur_entry >= limit ) { HeapWord * cur_hw = _ct -> addr_for ( cur_entry ); if ((* cur_entry != CardTableRS :: clean_card_val ()) && clear_card ( cur_entry )) { // Continue the dirty range by opening the // dirty window one card to the left. start_of_non_clean = cur_hw ; } else { // We hit a &quot;clean&quot; card; process any non-empty // &quot;dirty&quot; range accumulated so far. if ( start_of_non_clean < end_of_non_clean ) { const MemRegion mrd ( start_of_non_clean , end_of_non_clean ); _dirty_card_closure -> do_MemRegion ( mrd ); } // Reset the dirty window, while continuing to look // for the next dirty card that will start a // new dirty window. end_of_non_clean = cur_hw ; start_of_non_clean = cur_hw ; } cur_entry --; } … }
  35. 35. Patching OpenJDK Vicious loop - Optimized void ClearNoncleanCardWrapper :: do_MemRegion ( MemRegion mr ) { ... while ( cur_entry >= limit ) { HeapWord * cur_hw = _ct -> addr_for ( cur_entry ); if ((* cur_entry != CardTableRS :: clean_card_val ()) && clear_card ( cur_entry )) { start_of_non_clean = cur_hw ; cur_entry --; } else { if ( start_of_non_clean < end_of_non_clean ) { const MemRegion mrd ( start_of_non_clean , end_of_non_clean ); _dirty_card_closure -> do_MemRegion ( mrd ); } // fast forward via continuous range of clean cards // hardcoded 64 bit version if (((( jlong ) cur_entry ) & 7 ) == 0 ) { jbyte * cur_row = cur_entry - 8 ; while ( cur_row >= limit ) { if (*(( jlong *) cur_row ) == (( jlong )- 1 ) /* hardcoded row of 8 clean cards */ ) { cur_row -= 8 ; } else { break ; } } cur_entry = cur_row + 7 ; HeapWord * last_hw = _ct -> addr_for ( cur_row + 8 ); end_of_non_clean = last_hw ; start_of_non_clean = last_hw ; } else { end_of_non_clean = cur_hw ; start_of_non_clean = cur_hw ; cur_entry --; } } ...
  36. 36. Patching OpenJDK Serial collector gain http://aragozin.blogspot.com/2011/07/openjdk-patch-cutting-down-gc-pause.html
  37. 37. Patching OpenJDK CMS collector gain http://aragozin.blogspot.com/2011/07/openjdk-patch-cutting-down-gc-pause.html
  38. 38. CMS, PLABs and fragmentation
  39. 39. CMS, PLABs and fragmentation
  40. 40. CMS, PLABs and fragmentation
  41. 41. CMS, PLABs and fragmentation
  42. 42. JRockit’s gencon Hidden pauses
  43. 43. Tuning old space <ul><li>-Xms<N> -Xmx<N> </li></ul><ul><li>Young space is not for object, it is for garbage </li></ul><ul><li>If you old space is 100% full, GC throughput is zero </li></ul><ul><li>CMS: 50%-20% head room for GC </li></ul><ul><li>JRockit’s gencon / HotSpot’s G1: 50%-70% head room </li></ul><ul><li>Exploiting multiple cores (HotSpot JVM) </li></ul><ul><ul><li>-XX:+CMSConcurrentMTEnabled </li></ul></ul><ul><ul><li>-XX:ParallelGCThreads=<n> </li></ul></ul><ul><ul><li>-XX:ConcGCThreads=<n> </li></ul></ul>
  44. 44. OTHER ASPECTS OF GC IN JAVA
  45. 45. JNI and pinning objects <ul><li>Normally JNI code is working with copy of java object. </li></ul><ul><li>Native code my request direct pointer to java heap. </li></ul><ul><li>GetPrimitiveArrayCritical/ReleasePrimitiveArrayCritical </li></ul><ul><li>GetStringCritical/ReleaseStringCritical </li></ul><ul><li>While object is pinned, JVM cannot relocate it in heap. </li></ul><ul><li>Diagnostics </li></ul><ul><li>-XX:+PrintJNIGCStalls </li></ul>
  46. 46. Finalizers <ul><li>JVM have to keep special reference for object eligible for finalization </li></ul><ul><li>Object with finalizers survive at least one more GC cycle </li></ul><ul><li>Finalizer is called only then object is found unreachable during GC, GC of old space could be very rare </li></ul><ul><li>Use reference queues instead of finalizers </li></ul>
  47. 47. Soft references <ul><li>In server mode, JVM cleans soft reference if it was not dereferenced during N milliseconds. N is proportional to amount of free heap space after last old space GC. </li></ul><ul><li>-XX:SoftRefLRUPolicyMSPerMB=<N> </li></ul><ul><li>LRU policy relies on usage of reference get() method, not access to referenced object. </li></ul>
  48. 48. SURVIVING HUGE HEAP CONCLUSION
  49. 49. GC tuning summary <ul><li>Young space tuning </li></ul><ul><li>Find balance between, pause frequency, pause time (time to copy objects) and object promotion age </li></ul><ul><li>Old space tuning </li></ul><ul><li>Ensure throughput by providing enough head room for garbage </li></ul>
  50. 50. Surviving with huge heap <ul><li>CMS is very good in terms of pauses </li></ul><ul><ul><li>You can reliably keep pauses under 150ms – 50ms on 30GiB – 50 GiB </li></ul></ul><ul><li>Fragmentation treat </li></ul><ul><ul><li>Not big deal for server type of applications </li></ul></ul><ul><ul><li>XML processing is GC disaster </li></ul></ul><ul><li>Very narrow GC comfort zone </li></ul><ul><ul><li>If you tune for “long run” you are likely to have pauses during initial loads / bulk refreshes </li></ul></ul>
  51. 51. MOVING OUT OF HEAP
  52. 52. Direct memory buffers <ul><li>java.nio.ByteBuffer.allocateDirect() </li></ul><ul><li>Pro </li></ul><ul><li>Memory is allocated out of heap </li></ul><ul><li>Memory is deallocated when ByteBuffer is collected </li></ul><ul><li>Cross platform, native java </li></ul><ul><li>Con </li></ul><ul><li>Fragmentation of non-heap memory </li></ul><ul><li>Memory is deallocated when ByteBuffer is collected </li></ul><ul><li>Complicated multi thread programming </li></ul><ul><li>-XX:MaxDirectMemorySize=<value> </li></ul>
  53. 53. RTSJ <ul><li>Scoped memory </li></ul><ul><li>Objects can be allocated in chosen memory areas </li></ul><ul><li>Scoped and immortal areas are not garbage collected </li></ul><ul><li>Scoped areas can be release by whole area </li></ul><ul><li>Cross references between areas are limited and this limitation is enforced in run time </li></ul>
  54. 54. Unsafe java <ul><li>sun.misc.Unsafe </li></ul><ul><li>Unsafe.allocateMemory(…) </li></ul><ul><li>Unsafe.reallocateMemory(…) </li></ul><ul><li>Unsafe.freeMemory(…) </li></ul>
  55. 55. <ul><li>Thank you </li></ul>Alexey Ragozin [email_address] http://aragozin.blogspot.com - my articles

×