How we cut a 115-second sync to 4 seconds

Speed has been on our minds since April, when 19,552 of our own bookmarks froze the page. Our test library has since grown to about 40,000 bookmarks, so we measured the slow parts of Reading Rewind and fixed them, most of it in a single afternoon.

In our benchmark a full browser re-sync dropped from 115 seconds to 4.2 and importing 5,000 bookmarks went from 7.5 seconds to 0.8. The data sent when you click a bookmark shrank from 378 KB to 174 KB. Nearly every fix was a matter of doing work in batches or asking the database for less, and none of it needed new servers.

One bookmark at a time

A full re-sync used to look up and update bookmarks one at a time. Batching the snapshot import cut it from about 154,000 database queries to 824, and the time from 115 seconds to 4.2. Marking the bookmarks that vanished from the browser became a single update in place of a loop over every bookmark, and went from 3.1 seconds to 0.13.

The first sync of a new browser had the same problem with addresses. Resolving 5,000 new URLs took 10,006 queries and now takes 15, and that sync fell from 8.3 seconds to 4.7.

The extension was chatty as well. During a snapshot its side panel asked the server for status 91 times. Now it asks once.

The import, and a wall at 13,000

Importing a bookmarks.html file with 5,000 links took 7.5 seconds. We resolved all of its addresses in one go and inserted the bookmarks and tags in bulk, and the same import took 0.8.

A bigger import hit a wall. MySQL limits how many values one prepared statement can take, and somewhere above 13,000 new addresses our one big insert went over the cap. Address lookups and inserts now go in batches of 1,000, which keeps even huge imports and syncs under the limit.

Sync progress

While a sync runs, the app shows its progress. The old version streamed updates to every open app tab and kept a server worker busy for each one. Now the page checks in every 5 seconds and stops checking while the tab is hidden. Worker time per open tab dropped from about 54 seconds a minute to 0.2.

Lighter pages

The new list engine from the redesign gave us room to trim each page. Before and after:

• sorting All Bookmarks by date: 74 ms to 0.2 ms, using a stored sort column with an index
• a source's bookmarks page: 240 ms to 126
• a row click: 218 ms to 77, with the response shrinking from 378 KB to 174 KB
• the Cleanup Center's untagged list sorted by title: 190 ms to 70, plus two other Cleanup queries from 143 ms to 24 and from 130 ms to 12

On that row click, the quick-tag menu now travels as data instead of markup, and the tag options no longer re-render every time.

A bug we weren't looking for

One change made address lookups use only a hash of the address, and they went from 4.2 ms to 0.2. It also fixed a quiet bug. Before, two addresses that differed only in upper- and lower-case letters could overwrite each other's stored URL, and we only found out because we were chasing milliseconds.

Next: 18,918 orphaned bookmarks and the rule they taught us.