From d05bf044441dbf8e000036d545df9689bdec0b72 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Wed, 16 Oct 2013 14:03:22 +0200 Subject: [PATCH] Treat SQLITE_PROTOCOL as SQLITE_BUSY In the Hydra build farm we fairly regularly get SQLITE_PROTOCOL errors (e.g., "querying path in database: locking protocol"). The docs for this error code say that it "is returned if some other process is messing with file locks and has violated the file locking protocol that SQLite uses on its rollback journal files." However, the SQLite source code reveals that this error can also occur under high load: if( cnt>5 ){ int nDelay = 1; /* Pause time in microseconds */ if( cnt>100 ){ VVA_ONLY( pWal->lockError = 1; ) return SQLITE_PROTOCOL; } if( cnt>=10 ) nDelay = (cnt-9)*238; /* Max delay 21ms. Total delay 996ms */ sqlite3OsSleep(pWal->pVfs, nDelay); } i.e. if certain locks cannot be not acquired, SQLite will retry a number of times before giving up and returing SQLITE_PROTOCOL. The comments say: Circumstances that cause a RETRY should only last for the briefest instances of time. No I/O or other system calls are done while the locks are held, so the locks should not be held for very long. But if we are unlucky, another process that is holding a lock might get paged out or take a page-fault that is time-consuming to resolve, during the few nanoseconds that it is holding the lock. In that case, it might take longer than normal for the lock to free. ... The total delay time before giving up is less than 1 second. On a heavily loaded machine like lucifer (the main Hydra server), which often has dozens of processes waiting for I/O, it seems to me that a page fault could easily take more than a second to resolve. So, let's treat SQLITE_PROTOCOL as SQLITE_BUSY and retry the transaction. Issue NixOS/hydra#14. --- src/libstore/local-store.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/libstore/local-store.cc b/src/libstore/local-store.cc index 7f375097a..64b5dd961 100644 --- a/src/libstore/local-store.cc +++ b/src/libstore/local-store.cc @@ -48,7 +48,7 @@ static void throwSQLiteError(sqlite3 * db, const format & f) static void throwSQLiteError(sqlite3 * db, const format & f) { int err = sqlite3_errcode(db); - if (err == SQLITE_BUSY) { + if (err == SQLITE_BUSY || err == SQLITE_PROTOCOL) { static bool warned = false; if (!warned) { printMsg(lvlError, "warning: SQLite database is busy");