Hack week

Last week was Hack week at Novell. At the beginning of the week, I started working on my idea of a out-of-the-box backup/restore/archive manager application for Evolution. Due to some family occasion, I was out of station and hack-week, for three days. When I came back, I took up another idea - a continuation of my Evolution Exchange Performance work. Evolution exchange starts up in just 3 seconds (approx.) now. Patch is in nascent stage and more to come in another couple of days. Stay tuned!

Evolution Exchange Performance Series - Chapter 3 - Folder loading from 480 secs to < 1 sec

Close to a month now, have been working on squeezing Connector to save some time while it loads the folders. Remember Chapter-1 of my Evolution Exchange Performance Series blog - Yes, I based my optimization work from it. In the attached image, as you can see most of the folder loading time is spent in camel_exchange_folder_construct(). Construction of exchange folder objects does two major things:

• Scanning the entire folder for any changes since last evolution session and
• Fetch messages to keep things in sync

These two operations contributed to a big level in connector folder loading time. Fix doesn't refresh if summary is available and scanning for changes, since last evolution session, is moved to a separate thread with appropriate threading-etiquettes.

Here comes the supporting fact for my blog: :)



(2.11.x timing may vary depending upon production servers - response from my test servers are quite quick)

With this fix, Exchange Connector catches up with Microsoft Outlook w.r.t folder loading time. Outlook users will love Evolution 2.11.5 - watch out for more performance hacks. This fixes 442186, part of 341214 and 421091.

Evolution Exchange Performance Series - Chapter 2 - Public Folder Loading

Around a month back started working on issues surrounding Exchange Public folders from subscribing to accessing/using them. A tracker bug was already present in BGO and the most famous among them are 268412, 347811 and 346728. As I just took over the responsibility of Evolution Exchange connector around August 2006 and then had a month-long sick leave because of Chikungunya, it was a slow start for me - reading through code, docs, MS articles, webDAV references etc. After a week of analysis, found out the issue in Setting up folder hierarchies - which is done during the startup and it goes into a recursive loop setting up hierarchies for all folders, including public folders. When an organization has quite a lot of nested-public-folders, Evolution will take a minimum of 30 mins to start.

The complete fix was done in two-phases.
Phase-1:
--------
Do not scan public-folders recursively during startup.
Phase-2:
--------
Load folders on demand. This enabled us to achieve the maximum performance during subscribing to Public folders.
Any performance work needs to provide supporting data and here it is: (My test server has close to 250 public folders with a maximum depth of 21 subfolders)

S.No	Description of Test/task	Patch (Sec)	<=2.9.4 (Sec)
1	Configure an exchange account Restart Evolution	7	65
2	Folder->Subscriptions Selectthe configured Exchange account Note time taken to show the“All Public Folders”	3	4
3	Expand “All PublicFolders” Note time taken to display folder list	< 1	168
4	Time taken totraverse to maximum depth folder	8	330

Memory consumption:
Before expanding a folder (in the folder->subscriptions dialog)

	Patch (MB)	<=2.9.4 (MB)
Virtual Memory	152.8	154.6
RSS	13.4	15.1
Shared	9.1	9.2

After expanding a folder:

	Patch (MB)	<=2.9.4 (MB)
Virtual Memory	152.8	201.6
RSS	13.4	62
Shared	9.1	9.2

Actual memory consumption:

	Patch (MB)	<=2.9.4 (MB)
Virtual Memory	0	47
RSS	0	46.9
Shared	0	0

Note: Memory consumption data is taken using gnome-system-monitor and thus difference in Bytes are not shown above. Columns showing 0 would definitely have a difference at least in bytes.
Memory consumption to traverse to the maximum depth folder:

	Patch (MB)	<=2.9.4 (MB)
Virtual Memory	152.8	246.9
RSS	13.4	107.4
Shared	9.1	9.2

The fix also improved the overall folder loading performance.

Evolution Exchange Performance Series - Chapter 1 - Folder Loading

One of the critical issues in exchange mailer is - time spent to load the hierarchy or mail folders. See this graph:

It clearly shows most of time is spent in constructing folder objects. The sample size used to generate this graph is two folders with 210 and 50 mails repectively. Patch extends Federico's sample code and implements different levels of checkpoints. Graph on the left hand side is generated using a E2K_LOG_LEVEL=1.

As I mentioned in my previous blog, this is the customized plot-timeline script that generated this image.