Moving to a NAS-based storage solution

I have recently gotten a new Synology NAS (a DS214+ to be exact) with a 4 and a 6 TB HGST NAS to bring order to the chaos that is my data storage solution. Since I think I am not the only one, I hope my blog post (which is the first in a series of posts) will be of help to others who are in a similar situation and are mulling over what to get.

Over the years, storage has increasingly become a problem, a problem that was exacerbated by an upgrade to a Retina MacBook Pro with a mere 256 GB of internal storage and by the desire to share certain data with my significant other. My data was scattered across 3 internal and 5 external hard drives of various sizes, some of them containing duplicate data, and because Backblaze, my online backup service of choice, ignores Time Machine volumes for backups, not all of it was backed up twice.

I had tried earlier to bring some sanity to this chaos by buying a Transporter, but the software has never worked for me: just displaying remote folders would lock up the Finder with the spinning beach ball of death, and transfer rates were catastrophically low. Maybe the software works well if you use it as a private Dropbox, but I needed a remotely accessible network share instead. So it was clear I needed a different solution, and I thought of my different options.

My previous storage setup

Before switching to Backblaze, I used to use two external hard drives (3 and 4 TB in size) for backups, one located at work and another at home. One of them also stored my movie collection and my Aperture library containing older photos. I had multiple copies of my data, sometimes in the form of older versions of files and libraries (leftovers when I was migrating to new hard drives and such). Two of the three hard drives were filled to the brim. I wanted to simplify where and how I store files. Moreover, Backblaze will not backup volumes which also contain a Time Machine backup, and thus, the data on one of my drives is not backed up. In short, it's a mess accumulated over the years.

My wish list

(1) Backups
We have three (soon two) computers in our house, my gf has my old 2010 15" MacBook Pro while I use a new 2014 13" Retina MacBook Pro. We use Time Machine on our Macs, and we would like to store our backups on the NAS.

(2) Storage of movies (shared) I would like to share my movies with my girlfriend and have access to them when I am on the go.

(3) Storage of my big Aperture libary
My MacBook Pro's SSD sports a capacity of only 256 GB, too small to even just hold my photos. So I need relatively quick transfer rates to be able to use Aperture with photos that are stored on my NAS rather than my internal SSD.

(4) Access to my files from across the world
It would really be great if I didn't have to bring my external drives along when I travel, and I could access my data from anywhere.

(5) My own unlimited Dropbox
I would like to keep my Documents folder synced just like a Dropbox would. That includes full access via the web.

(6) Integration with online backup
A NAS is just a sophisticated enclosure for a hard drive. They are not a backup. They may be part of a backup strategy, yes, but if your only copy of a file is on your NAS, you have a problem. That's why I've been using Crashplan and Backblaze, and I would like to have an offsite backup of all the files on the NAS.

Enter the Synology NAS: two bays filled with HGST goodness

Given my capacity needs and my budget, it was clear that I needed to look into two-bay solutions. As far as quality brands go, you mostly hear two suggestions if you are willing to spend some money, QNAP and Synology. I decided to go with the latter, because Synology seems to have the best software support (from Synology directly and the community at large).

The amount of choice is a bit dizzying: just Synology alone offers no less than 6 (!) two-bay NAS solutions, some of which are at the same price point. For instance, the DS214play sells for the same amount as the DS214+, a little more transparency and less choice couldn't hurt. Eventually, I decided in favor of the second model from the top because of its beefier hardware and the second GBit connector. Moreover, the DS214play which features a more powerful Atom CPU and a hardware transcoder for various movie formats did feature the cheaper DS214 (non-plus) built quality.

Now on to the hard drives: I have dedicate one drive to backups and another to network shares. Capacity-wise I opted for a 4 TB backup drive (I need to backup about 1.4 TB, so that should leave ample headroom for older snapshots) and a 6 TB drive for network shares because that was the largest capacity that is affordable -- the 8 TB drives are just too expensive. For the longest time, I haven't much paid attention to the brand of drives I was getting, because for years I was just putting them in external enclosure where performance is limited by the interface for the most part. Fortunately the folks at Backblaze (see also here for the original blog post and here for the latest update) have analyzed the failure rate of hard drives, and found out that by a wide margin they found Segate drives to be the least reliable drives while HGST's are the most reliable. The most telling graph to me is this one:

36 month survival rate of hard drives by manufacturer

Over the course of three years, 26.5 % of all Segate drives fail! Ouch. At least 2 of my 4 external hard drives use Segate drives, so I am glad I am relegating the external drives to a higher-level backup. Of course, there is variability among models, and with newer models the difference isn't as large as before (although it is apparently still a factor of 2 or larger). Moreover, drives which spend their time in one of the Backblaze storage pods will have much more wear and tear than those in my NAS. Nevertheless, Backblaze's data is a clear indication that Segate drives are least reliable and HGST drives are the most reliable. And the difference in failure rates is significant enough to me.

So without looking at benchmarks I had decided on HGST's NAS line of drives. Curiously, the 6 TB model has a 128 MB cache, about half of the total capacity of my first hard drive. That gives me about 10 GB raw storage in total.

First impressions: hardware

The built quality is exquisite: the case is all metal, the hard drive trays are tool-less if you use 3.5 inch drives and slide into place without so much of a sound. Even the power brick feels much more substantial than the POS that is usually included. Very well done, Synology. The hard drive bays are hot-pluggable, so I did not even have to power down the unit when I installed the 6 TB hard drive which arrived a few days after its smaller 4 TB brother. Everything you need (e. g. screws and two network cables) was included in the box, also here a big thumbs up. The unit is relatively quiet, I can only hear the hard drives's platters spinning and heads seeking. The HGST drives I chose sound particularly crunchy and crisp. On the bright side, I cannot hear any fan noise if I put the unit in quiet mode (which is the default).

The only real point of criticism I have is that the CPU (a dual-core Marvell Armada XP MV78230 running at 1.33 GHz) is quite slow compared to the competition, QNAP just announced a unit with an Annapurna SoC that sports two Cortex A15s. More on that below.

First impressions: software

The main advantage of going with an off-the-shelf NAS from manufacturers such as Synology and QNAP is actually the software. You can approximate the hardware yourself, even though whatever you build will be larger and less elegant, you'd probably end up with more powerful hardware as well as a more flexible system. But really, the main selling point to me is the integration of software, hardware and services.

And here, the Synology does not disappoint. Overall, you get a OS GUI-type interface that is delivered to you via html5. I reckon this cuts down the bandwidth requirements significantly compared to a proper VNC solution. The UI is a bit of a mongrel combining elements of Windows and OS X, while the lack of visual polish feels very Linux-y. Certainly, I don't care as much about visual aesthetics in a UI that I'll use sparingly. But is it functional? Yes, the UI is quite fluid, and the settings are grouped logically, and if you change one thing that relies on another, you get sent to another settings page automatically. The UI also prevents you from essentially bricking your system, e. g. at one time when I was switching the firewall on and off (I had to troubleshoot the VPN), I forgot to toggle between »disallow all« to »allow all« after disabling all rules. Sure enough, the settings app wouldn't let me save the configuration and explained very clearly why. Nice. The UI comes with Windows file sharing enabled, and configuring AFP is as simple as ticking a checkbox.

Setup

First setup is quite straightforward in a regular network where the system is assigned an IP address via a DHCP server and you have a connection to the internet. The procedure is straightforward, and I won't detail it here, but the only decision you absolutely need to make yourself is how to configure your logical volumes. I went with one volume per disk because that fit my use case and budget best, but that doesn't mean it's the best for you. The Synology suggests you to make a surface scan on each drive, I quite like that.

Configuring the default services

The defaults are very sensible, after creating a shared volume and a user, you have an SMB network volume ready to go. The more advanced functionality is easy to get to if you understand how networked services work. And even if you don't, Synology tries to lend a helping hand. E. g. if you give it login credentials to your router, it'll automatically create port-forwarding rules. While my router is listed as being supported, this actually did not work for me. But seeing how, ahem, questionable my router's software is at times, I don't think it's quite as simple as blaming Synology.

Also being able to access your files from outside of your home's network is very easy. You essentially register your Synology DDNS service via your NAS's interface. I wouldn't believe it at first, but it just worked. With portforwarding set up manually, I can configure my NAS from any place with an internet connection. You should choose https, though, so that no one who isn't supposed to be peeking into your traffic will have access to your NAS. In principle this also allows you to configure other services such as VPNs. You could open the file sharing service's ports, but you shouldn't. That's what VPNs are for. More on that below.

I could also configure my VPN to take over most of the functionality that I leave to my router (well, routing, DHCP server, firewall, etc.) -- especially since it comes with two GBit ethernet ports. But I don't think it's a good idea to leave my NAS exposed to the raw internet like that. Moreover, you can use it as a print server if you want.

Adding functionality via packages

I haven't counted the number of packages, but they cover the most common use cases such as a VPN, a virus scanner, media server. Some models also offer a Plex server, but sadly, Plex doesn't work on my DS214+. It works on the 213+ and the 214play, but not on mine. Ugh. The virus scanner seems to really push my system, it'll quickly devour half of my RAM and most of my CPU cycles. Nevertheless, the packages work well. Only the media server won't really play ball, but on the other hand, I can play videos from the mounted volumes just as well. I ran into trouble configuring the VPN, and I am not quite sure whether it's my ISP who doesn't want to play ball and allow their customers to create VPNs or if it is a bug or user error.

Problems and complaints

I ran into a few isolated problems. For instance, I noticed that on large copy operations (which you then leave to complete in the background), the status bar in the UI will freeze. Nevertheless, the copy operation is carried out and you will eventually get a notification. Only the first time did I abort the copy operation by hand and start it again. Some of the settings seems kinda fiddly, e. g. in some interfaces, you have to deliberately press save buttons, otherwise your changes are lost.

Missing features and downsides

(1) A modern filesystem
My data is very precious to me, and I would prefer a filesystem that does checksumming, allows for snapshots and features copy-on-write instead of ext4. In short, I would have preferred OpenZFS or btrfs, and since OpenZFS is more mature (it has been in development since 2001) and I am more familiar with it, I would hope for the former. Netgear's newer models use btrfs instead of ext4. I am familiar with ZFS a little, I ran it on my Mac for a while. I would have loved to have more than 1 copy of important files. Moreover, given ZFS's focus on filesystems instead of directories, I could set quotas and such on a per filesystem level instead of the user level. I know that these more modern filesystems come with a healthy appetite for RAM and cpu performance, but these are problems that time eventually will take care of.

(2) More powerful system specs
I am quite glad I got a model with 1 GB instead of 512 MB RAM, because even the few services that I run on my Synology at this time, I use about 50 % of the RAM. What is more, normal operations can easily suck up 50-80 % of the CPU power, and activating the anti-virus software maxes out the CPU. That leaves virtually no headroom, and makes me a bit uncomfortable. The competition is switching to more powerful SoCs, e. g. Annapurna Lab's new Cortex A15-based solutions which are -- thanks to hardware accelerators -- very, very fast. The higher-end models feature two 10 GBit ethernet and 4 cores and outperform comparable x86-based models. Synology is a bit behind here.

(3) Better damping of the hard drive trays
Sometimes I can hear quite annoying harmonics when both drives are active. While the drive tray feature rubber rings around the screw holes to dampen vibrations, they are not as effective as they should be. I hope Synology takes another crack at it and adds more rubber to the drive tray.

(4) ECC RAM
Again, I love my data and the use of ECC RAM would give me a bit more peace of mind. But then again, I know ECC RAM is right now out of the price range for this class of device.

(5) The software needs more polish
At a few places, the software needs a bit more polish. For instance, the Cloud Sync Utility at one point hogged my cpu, it seemed as it was in competition with Dropbox syncing things back and forth. Likewise, the NAS's interface has a few quirks (e. g. the use of save buttons at times) which should be cleaned up.

Managing transitions and expectations

Recently, there have been increasingly loud voices criticizing Apple's software quality, and some concerns whether Apple has abandoned the »pro« market. Articles and blog posts centering around these themes seem to ebb and flow in regular intervals.

While I do think there are often good particular reasons to make any one argument, it is not very productive to look at them in isolation. For instance, is Apple abandoning development of Aperture a sign that Apple no longer believes in serving the »pro« market? Or is it just one incarnation of a larger issue inside the company?

Moments of transition

With the announcement of Windows 10, Microsoft has embarked on a journey from a desktop PC-centric approach to a world where for each person you have several devices. And interestingly, also another major piece of Microsoft software is going through a chrysalis with some awkward consequences. I'm speaking of Office: it was Office for the iPad which showed the way. Based on a common core with all other versions of Office, it showed off the new design language Microsoft intends to use. Also a lot of their other pieces of software are rewritten so that one universal app works across all devices.

That sounds familiar, doesn't it? Just like Microsoft Office is going through an awkward phase, also Apple's office suite is going through a similar transition: it cut features on the desktop version in favor of starting afresh with a codebase that is shared amongst the iOS and OS X apps. There are other victims of this period of transition, Apple's iPhoto and Aperture come to mind. Here, the motivation is not so much to use a shared codebase, but rather a tectonic shift in the philosophy. iPhoto and Aperture were born in the digital hub era where the personal computer, the Mac, is in the center of your digital life and contains »the truth«. With the introduction of iCloud, Steve Jobs pronounced the end of the digital hub and put the cloud in its place. Step-by-step Apple has been busy updating existing apps such as iTunes to take care of »the cloud«. Clearly, at a certain point in times it would have been iPhoto's and Aperture's turn. While I don't think it was wise to cancel Aperture before a similarly powerful replacement was ready, more on that below, at least on the grander scheme of things the direction makes sense.

From simple to complex

From a glass half-empty perspective, people on the desktop are sacrificing functionality in favor of a platform they may not even use. Apple's focus primarily lies on iOS rather than OS X, and this is clear evidence of that. An optimist would counter that this is a chance to re-evaluate old pieces of software with old code bases, and to make them more user friendly. Microsoft and Apple took the opportunity to start from scratch.

I find this trend quite interesting, because it reverses the direction in which features have usually flown: in the past, professional, complex apps were often the starting point for simpler, less powerful apps. Photoshop gave birth to Photoshop Elements. Apple derived Final Cut Express from Final Cut Pro. iPhoto eventually inherited the database format Aperture was based on.

Nowadays, more powerful desktop software is reimagined after having to design software for tablet and phone operating systems. Microsoft Office for Windows 10 is based off on Office for the iPad. The restrictions force software designers to reevaluate their decisions, and they often find that simpler interfaces are more appealing than being overwhelmed by choice. I see this as a huge opportunity to make desktop software more user-friendly, and the personal computer as a whole more humane.

Cross-device interoperability

The trend also goes to people having several computers. Even thermostats or lightbulbs are based on computers. So a lot of efforts are going into integrating them seamlessly between devices. Microsoft will allow you to stream XBox One games across all your Windows 10 devices, meaning you can continue to play on your Suface or PC in case someone else wants to watch TV. Apple's Continuity features enable users to »move« tasks across devices. The lines between the different computers are blurred. And most of these features are relying on the cloud. This is the major push behind all these moves.

Managing software transitions, managing expectations

Apple has a mixed track record when it comes to software transitions: they have excelled in some (e. g. migrating from classic MacOS to OS X and from PowerPC-based Macs to Intel-based Macs), more recently, their record was much more, ahem, mixed. The first botched transition was that of Final Cut Pro to Final Cut Pro X. While I am not a video editor, and I am not fully qualified to judge, it seems to me that a major flaw was Apple's management expectations. I am fairly certain that if Apple had renamed Final Cut Pro X into Final Cut Xpress and announced that future versions of Final Cut Pro will be based on it, the reaction would have been much more calm. Instead, a vocal part of the Final Cut community got their pitchforks out and flailed Apple. Also with Aperture, I think it was a big mistake to half-heartedly continue development of one of the most popular paid apps on the Mac app store, and discontinue it before a suitable replacement is ready.

Lack of light tower apps

What is really disconcerting to me is the scarcity of Apple light tower apps, meaning apps where Apple shows off what in their view a good Mac app should look like and fulfill a need. A long time ago, these light tower apps were the iApps on the consumer side as well as pro apps such as Keynote, Final Cut and Aperture. I don't think the significance of these apps can be understated for people who switched to the Mac before it became en vogue to do so. These apps were revved regularly, and often in lock-step with the OS. However, the UI of many of these apps got worse (the most glaring offender here is iTunes, but also iPhoto got interface elements which seem very un-Apple-y).

And it's not just the Mac which seems to be suffering, it is iOS, too. Right now, iOS hardware seems overpowered for the type of apps iPads and iPhones need to deal with. For comparison, an iPad Air 2 from 2014 is roughly as powerful as a 2011 11" MacBook Air. iOS also seems to have all the plumbing to make it very powerful. What is missing are Apple apps which take advantage of this. They've tried with iPhoto for the iPad, but it is safe to say that this experiment has failed. If Apple subscribes to the vision that iOS devices will supplant traditional PCs for most tasks, they need to give us apps which do things for which right now you still need a PC.

Software quality issues

While I don't subscribe to the hysteria that Apple is doomed, I think software quality across the board is an issue: it starts with bugs in the OSes but continues with rare updates to popular apps. Hardware-wise Macs are best-of-breed, and Apple has consistently anticipated trends correctly. So the obvious lack of finish on its software products is even more glaring. From the outside it looks as if Apple is constantly redlining its software team, so that major pieces of software (including consumer-grade software such as iPhoto) are no longer updated on a regular cycle. It's part of the failure of its software division to keep up with the growth of its hardware division. According to me, Apple's biggest challenge is not to enter new markets, redefine other categories or some such, it needs to bring its software to where its hardware is.

The TeX-Wrangler: arguing with the arxiv

People who like to use some of the more advanced LaTeX features sometimes run into problems when trying to put things on the arxiv. The arxiv uses a relatively old LaTeX distribution (TeX Live 2011) – which is a big upgrade from TeX Live 2009. If they continue the update pattern, they will adopt the latest TeX Live distro every other year.

Hence, if you just install the latest TeX Live on your Mac or Linux machine, some packages cause trouble. One example is the biblatex + biber combo which is a replacement for bibtex. Two reasons made me switch away from bibtex: bibtex is not good at handling non-ASCII characters such as umlaute (ä, ü, ö) and accents (e. g. é and à). The second and more important reason is that it is hard to customize the output of bibtex to your liking. As the arxiv does not run bibtex or biber on the server, the bibliography needs to be created on the client computer. 

Biblatex and biber have seen a major upgrade from TeX Live 2011 (v1.7) to TeX Live 2012 (v2.6), and it turns out that the output is mutually incompatible. On the other hand, biber had had some trouble handling large bibliography files on OS X, something that has been fixed in the latest release. Even though I have tried to include all necessary files, in the end, I gave up and switched back to TeX Live 2011 on my machine just to process this file using the old version of biblatex and biber so that the output is compatible with the arxiv. Fortunately OS X allows you to easily switch between the different TeX Live distros in the System Prefs. I have also had to shorten my bibliography, but fortunately that is something that can be automated using BibTool, for instance. 

So if you have trouble submitting things to the arxiv, install TeX live 2011 on your computer and switch to that once you are ready to submit to the arxiv. Then submit.