Fixing the Firefox Sync Server

Permalink

UPDATE 2020-05-09: added at the end a perfect deploy guide and a bit of config tunning for the deploy. I have the guide archived just in case it’s lost. The small-sized guide can be found at the end of this post.

I changed from Chrome to Firefox a long time ago, and I mostly don’t regret it. I prefer to have all my personal data under my control, and that is the reason I host my own Firefox Sync Server.

Chrome, of course, has your data within Google’s servers (the botnet). Firefox, too. But, at least, Mozilla has a 2018-something version of the synchronization server open in GitHub (Firefox Sync Server).

I upgraded the software a month or so ago. I though it was easy, just following the steps in the README file and it’s done. And, of course, it works. The software is upgraded after doing those steps. The problems are the changes made and not described in any part but a lost issue in the repository issue tracker.

This week I noticed that my changes weren’t synchronizing between my two computers. I though it was a problem of the Nginx configuration, so I started digging into it. It wasn’t. Logs didn’t show anything special but one thing (paste):

1586625940767	Services.Common.RESTRequest	DEBUG	GET https://DOMAIN/token/1.0/sync/1.5 500
1586625940767	Services.Common.TokenServerClient	DEBUG	Got token response: 500
1586625940767	Services.Common.TokenServerClient	WARN	Did not receive JSON response. Misconfigured server?
1586625940767	Services.Common.TokenServerClient	DEBUG	Content-Type: text/html; charset=utf-8
1586625940767	Services.Common.TokenServerClient	DEBUG	Body: <html>
  <head>
    <title>Internal Server Error</title>
  </head>
  <body>
    <h1><p>Internal Server Error</p></h1>

  </body>
</html>

1586625940767	Sync.BrowserIDManager	ERROR	Non-authentication error in _fetchTokenForUser: TokenServerClientServerError(...)(resource://services-common/tokenserverclient.js:39:36) JS Stack trace: TokenServerClientServerError@tokenserverclient.js:105:16
_processTokenResponse@tokenserverclient.js:279:19

What the fuck?!

I started diggin under the skin during 2 days until, for some reason, I left the watch command open with the status of the systemd's service and found this (paste):

ProgrammingError: (psycopg2.errors.UndefinedColumn) column "keys_changed_at" does not exist
LINE 3:     keys_changed_at, node
            ^

[SQL: select
    uid, generation, client_state, created_at, replaced_at,
    keys_changed_at, node
from
    users
where
    email = %(email)s
and
    service = %(service)s
order by
    created_at desc, uid desc
limit
    20
]
[parameters: {'email': u'REDACTED@api.accounts.firefox.com', 'service': u'sync-1.5'}]

Now, you might have guessed I use PostgreSQL to store the data. From my old setup to the new one, even doing the steps of the docs, the database wasn’t upgraded at all. For this, I spent 2 days looking into the wrong place. The fix is in an issue (you can go to the issue), but if you don’t want to get into it, here’s the fix:

ALTER TABLE users ADD keys_changed_at BIGINT;
ALTER TABLE users ADD node VARCHAR(255);

Where do you have to apply it? Of course, in the database you put into the configuration. And, with such simple thing, it works. It could be pushed as a migration to be done in the make build step or something, but nope.

I faced another problem related to the Nginx configuration I have in place, but that’s not relevant for this post.

And that’s a writeup made by spending two days fixing something the upgrade process should consider in the documentation. I understand that hosting yourself the software has its problems, but damn…

There’s a lot of things to improve with the Firefox Sync Server, for example, not using Python 2.7 or a better documentation, because it’s not clear at all.

Here’s a small thread about my frustration. I intend to continue from there to post my frustration with this piece of software…

If you want to deploy a Firefox Sync Server in your own server, or VPS, or whatever server you own, a guide to deploy the Firefox’s Sync Server can be found at Firefox Bookmark Syncing (Wayback Machine backup), writen by Ansgar Kellner. Credits to him, it works like a charm.

The guide above this paragraph doesn’t take into account using PostreSQL as the backend database, so you must do this in order to install witk before executing the make build command:

$ echo "psycopg2" >> requirements.txt

If it fails, try with psycopg2-binary instead.

Then, I recommend to you to setup the server for using gevent workers. To install them before installing the sync server:

$ echo "gevent" >> requirements.txt

If you have deployed your sync server already, you can install it and setup in the config file:

$ pip install gevent

In the file syncserver.ini:

[server:main]
...
worker_class = gevent

The three dots represents the rest of the configuration, put the last line at the end of that category.

And, with all of this, you’re going to have a smooth and fast Firefox Sync Server at your disposal.