Author Archives: Jonathan

Backblaze one year review: 70% data loss unexplained, mediocre software, distrust

tldr;

This article is a review and record of my unfortunate experience with Backblaze Personal Backup service, software, and support. If reading quickly, please go down the page to read the conclusion first and then any other sections after. It will take approximately 18 minutes to read the entire article.

About Backblaze

As an IT professional, I am regularly called upon for my opinion on both personal and business services that need a reliable solution and accordingly, when I get the opportunity to try out a service, I like to take advantage of the chance and have personal experience with the product to make the best recommendations later.

Backblaze claims to offer, “Never lose a photo, video, or file again,” “Cloud backup made easy and automatic,” and as a special offer, offered a six-month Backblaze trial included February 2017’s Humble Bundle. Having made use of Backblaze’s hard drive data in the past, I thought this would be a good occasion to assess their service and platform. When the six-months passed, I elected to pay for an entire year to comprehensively review.

PC Specifications

Backblaze and hard driveThe PC I have been running Backblaze on is using Windows 10, an i7-4790K non-overclocked CPU, 32 GB of RAM and six hard disks comprising 6 TB of data total. The disks use NTFS, MBR, separate drive letters and are in a non-RAID configuration. Two of the disks are 500 GB SSDs and the others are HDDs of assorted sizes.

Disaster strikes

At the end of January, I was moving data from one hard drive to another. During the move, the transfer speed suddenly dropped, and a dreaded cyclic redundancy check error appeared. After a reboot, the disk no longer mounted, and I quickly realized it had failed completely.

I had noticed that the same failed disk contained some bad sectors just a few months ago, so this failure was not a shock.

Now with 1.5 TB of data failed, this was my occasion to try a full drive restore with Backblaze. With a new hard drive ordered, I set off to the Backblaze website to get the restore process underway.

Trying to restore data

To restore data from the web on Backblaze, one selects folders to produce a non-compressed ZIP archive, up to 500 GB at a time. To start restoring my disk’s files, I selected about 70 GB, and let the server to produce the archive.

Uncharacteristically quick, I received an e-mail indicating that the restore archive was ready to download, but I immediately noticed that the size of the archive was only 1.8 GB. Downloading the archive confirmed that most of the files were missing.

As there had been no error message to suggest a problem, I attempted to restore a single file that was appearing on Backblaze but missing in the archive. Within seconds of attempting the single file restore, I received an email with the subject: “We weren’t able to complete your restore request” and on the Backblaze website, the restore showed as Failed.

Backblaze: We weren't able to complete your restore request.

Backblaze showing failed restore

Dealing with support

Prior to this event, I had previously sent Backblaze seven support requests. In one request, I experienced the upload process being stuck on “Producing file lists”, in another, files were not showing up on the website despite being uploaded (Backblaze says it takes 48 hours, in reality it can take weeks), and a feature request for the expected behaviour for a notification area-based (more about that in the software section below).

In my penultimate request in December, I described how after a reboot, the software had appeared to have lost its place, starting the amount of uploaded data from zero, and appeared to be re-uploading files that were already there. Backblaze’s response was that this behavior was due to a “self-healing” process and after uploading pre-existing files for about a day, the software seemed to be back on track. You will hear more about “self-healing” in a moment.

To be concise, the below does not include all the text of my support request. You can click on the hyperlinks in each section or see all the support tickets on my other post.

I began my request by explaining my situation:

But the restore [archive] is only 1.8GB, and in the zip archive most of the files are missing. I selected some of the files that are missing manually, and the restore took only a few seconds to prepare and shows up as “Failed”. Is there anything I can do?

I received a response 12 hours later:

In order to see what’s going on with your backup we’ll need to get the password for your account.

I am pleased Backblaze needs my password to get into my account, but I would hope they could already look at their server logs and see the errors. But regardless, after changing my password, I provided more insight and exact examples of the missing data:

I’ve done more attempts at restores, and for the most part, no restores seem to be complete. In one case I selected a whole group of files at 75GB, and got back a 128KB zip with a bunch of text files. Trying to restore individual files from those folders results in a ‘Failed’.

As an example, if you attempt to restore T:\Lorax, which is an old video project of mine I did for one of my sister’s plays. The selected files are 8,344.64 MB, but the resulting zip is only 1.55GB, and attempting to restore the main file (also missing from the zip), T:\Lorax\Lorax Youtube 1080p.mp4, results in a Failed response.

Another example, T:\TapesEdit, which are my childhood cassettes I digitized and edited at one point. Selected: 7,409.11 MB, but the resulting zip is 463.7 MB. Attempting to restore one of the missing files in the zip like T:\TapesEdit\Flubber.wav, results in a Failed [notification].

Five hours later, I received a response:

After further review, it appears that several large files in your backup from the T drive are missing parts from their file. Typically when this happens Backblaze will go through a process called self-healing where the missing parts of a file will be reuploaded. However in your case it looks like 2 files from the 2 drive were never able to be backed up at all for some reason which kept the backup in an initial backup state. The self-healing only occurs once the initial backup is complete, so because your backup never left the initial backup state the self-healing never occurred to fix these files in your Backblaze backup.

I had noticed that the software was stuck on two files before (or maybe 508? Or 482? More on that shortly) but had yet to open a support case for that problem. The software gives no indication of why it would be stuck in this way, and no error message appeared. I would not have known it was “stuck” if I didn’t often open the software regularly.

As mentioned before, I had seen this “self-healing” in the previous request. However, they go on to say there’s no hope for my files (emphasis mine):

Unfortunately because of this we do not have the several files from your T drive in your backup and there is not a way for us recover them as good versions of the files were never uploaded to our servers successfully. What appears when you download folders from the T drive now is the entirety of what is on our servers for the drive. This was a failure of the Backblaze system to work correctly with your computer and drives and if you did want to cancel your Backblaze service we would be happy to offer a full refund of your subscription. If you would like to continue with us then a fresh backup would be necessary to make sure that files are all backed up correctly.

The “several files” amounts to hundreds of files and at least 143 GB with more soon to be discovered. This answer sounds like they have done little investigation and are solely relying on the information I gave them and somehow blaming my “computer and drives” without even asking for the Backblaze software log files.

Additionally, if my options are to get a refund or start a fresh backup, why wouldn’t I take the refund and just sign up again? Furthermore, despite a major data loss, there is no further examination, no detailed description of the events leading to the failure, and no explanation of why this would not happen again.

Not being satisfied with this answer, I started doing my own research. After some testing, I found that the data loss affected all drives, not just the one that failed:

I did some further tests tonight, and the data loss is not limited to the T drive. For example, trying to recover a folder of 13.4GB from the D drive, results in a 43.08MB zip file. In addition, I did manage to determine what’s left of T on Backblaze and I’ve found that only 423GB is still on the server, which means 1,103.12 GB has been lost from T.

I also mentioned that I had seen the self-healing previously, providing the support request numbers back in December. Next, I looked at the software’s log files to see if I could figure out why the two files were stuck, particularly since Backblaze insists they are the root cause of why the data had gone missing:

Looking again through bztransmit23.log, I see “ATTEMPTING TO DECLARE VICTORY on initial upload” and then “BackupSummaryUserWouldSee=Selected_1,508,660_files_/_6,028,700_MB__Remaining_508_files_/_3,728_MB
NOT DECLARING VICTORY on initial upload – too much left”
However, the remaining files that are noted in the log never goes lower than 482, but the UI only shows two (and you also mentioned that there’s two).

I have zipped/attached my full logs to this message and hopefully you can figure it out better than I.

Since they did not request my log files, and their explanation seems to be based on the backup status screen on the website, I decided to attach them on my own, hoping for a better explanation:

Is there any explanation for how this happened and why it won’t happen again? From my perspective, this plane fell out of the sky, half the people died, and the solution is to build another one and hope for the best? I already know that at least a terabyte is gone and I’ll gladly provide any info or data you need to better analyze and figure this out, but from this side, it seems like you guys don’t care to know or you do know and it’s some sort of secret.

Finally, I revealed that I had not exclusively trusted Backblaze with the only copy of my data and that I wish to see their self-healing in action:

I have another backup of the T drive here, which was done the night before this all happened and I’ve restored it to a new disk tonight. With my data back, I would like to test whether this self-healing solution will work.

Three days later I received a response (emphasis mine):

Your description of self-healing is correct – large files from your backup on our servers are missing chunks of the file and Backblaze goes through the self-healing process to correct this and make files available once more via the Restore browser by attempting to reupload the missing chunks of the file(s) in question.

My request to know more about the stuck files was ignored nor does it seem that there was any further analysis from my attached log files:

We can’t say exactly why this happened but it’s likely that either [sic] the drive had been malfunctioning in some capacity for some time that could have caused this.

This does not make any sense as most of my files were uploaded prior to this hard drive problem, and this massive loss of data affects multiple drives. Moreover, if one drive failing can destroy all your uploaded files, doesn’t that defeat the point of the backup?

The self-healing is attempting to complete but has not been able to and will not be able to because of the drive failure. I’m glad to hear you were able to recover the data off the T drive. The only way to test this would be to start a new backup which will rebuild the data files that keep track of the hashes of these chunks to prevent this from happening in the future.

I had restored all the data to a new drive using my own backups, which they acknowledge. However, in their prior response, the only thing preventing self-healing was two stuck files. But now they claim that self-healing won’t work, and I need to start again, even with the stuck files no longer being an issue. Once again, their explanation does not make sense.

Furthermore, how would you restart your backup from the beginning if all your data was already lost by Backblaze?

This final message came on a Saturday, and by this time I had already concluded that the data loss affected every drive, and nearly all the data was not restorable on Backblaze. By Monday, they had automatically closed the ticket.

There has been no follow-up, no investigation, and no further communication from Backblaze since.

Backblaze made it precisely clear how much they value a customer and their data, and that is the full value of a refund, exactly 50 US dollars, if you ask for it.

How much data lost?

I meticulously went through all six of my drives, noting what Backblaze’s website thinks the size of the restored files should be and then compared the actual size once after requesting the files.

I had 5884.73 GB backed up to Backblaze, and of that, only 1683.98 GB was recoverable, with 4200.75 GB lost. That is 71.38% of all my files that are gone without explanation.

Were the files there before?

I was curious if Backblaze had lost my data after it had been on their servers for a time or if the files had never transferred properly in the first place.

Thankfully to answer this, I had previously made use of my Backblaze account to quickly transfer several single files (500 MB and 3 GB respectfully) and one large folder of Messenger install files (6.3 GB) by restoring to a remote computer that others could download from. An attempt to restore the single files was completely unsuccessful and the folder (previously successfully restored on December 7th), also failed as only 176 MB of the data now remains.

Both the files and the folder that previously restored successfully and now are unrecoverable, were not stored on the drive that failed.

This is confirmation that files previously uploaded, stored, and fully restorable have been since damaged on Backblaze’s servers and lost permanently by the company.

Backblaze Software

There is a lot on the surface that could be discussed regarding the Backblaze software: a strange drop-down menu that requires you to hold down the mouse to select anything, no keyboard support in the main Control Panel, backup exceptions that require editing of a text file to include a drive letter, and the lack of being able to back up files that are open/locked by Windows.

However, as Backblaze insists the software caused the loss of my files, let’s review some of the fundamentals of the software, as it does not follow the normal software design/engineering rules of Windows applications, and the result is confusion, strange behaviours and significant performance problems.

Notification Area/System Tray icon

Now and then, the Windows taskbar/explorer needs restarting manually, or it restarts on its own. When this happens, the applications that use the notification area (also known as the system tray, by the clock) need to re-add their icon or the icon will no longer appear. Mercifully back in 1998, Microsoft added an easy and uncomplicated way to re-add the icon, and I remember as a kid being particularly excited, rapidly adding this feature to all my self-made programs. Today, all Windows-supporting frameworks have this function as a feature that is built-in.

But when I started using Backblaze, any time Explorer restarted, the icon for Backblaze in the notification area would vanish. Eventually I got annoyed enough and sent their support a request, along with a Microsoft developer link to explain how they could add the feature into their software. The next day I received word that they had made this change and shortly after an updated version of Backblaze arrived with a restore-able notification area icon.

Although I was pleasantly surprised this was added so quickly without further fuss, realistically all Windows software that operates in the notification area has been expected to do this since 1998.

Missing version, description and information resource

Backblaze’s software runs multiple processes and updates them to new versions often. Do you want to know what those processes are, or what version you’re using? Don’t plan on opening up the properties of the files, or checking Task Manager to find out, as none of the Backblaze programs have any of the expected information (known as a VERSIONINFO resource in Windows).

(For current users: the About window hides in the menu on the notification area icon.)

Loose Threads

Modern software uses a concept known as threads to perform many actions at once, and for the scenario of backup, you use threads to maximize the speed of transferring data by sending multiple files (or chunks of files) all at the same time.

Backblaze offers a non-optimal form of threading for both uploading and downloading. For uploading, Backblaze threading is enabled by default and it would be painfully slow otherwise, as the software divides up files larger than 30 MB into small 10 MB file chunks and then uploads each chunk separately.

However, Backblaze’s concept of threading on Windows is employed poorly. Unlike Unix-based/inspired operating systems like Mac OS or Linux, implementing threads by starting new processes in Windows is performance-intensive and should be avoided in most situations. Instead, threads are expected to be implemented in the same process the program is running in.

The Backblaze software has been designed against these Windows fundamentals: for uploading, they have 21 identical “transmit” executables, named sequentially from 00 to 19, and the ‘original’ bztransmit.exe. Then all these files are duplicated with 64-bit versions.

Backblaze threading files

For the Backblaze downloader tool, the files follow the pattern with the tool duplicated 13 times for 13 threads:

During the upload process, these thread [worker] processes start from nothing, close, start again, with every 10MB file chunk they transfer, writing and reading a different configuration file each time, thereby unnecessarily reducing computer performance. Using Process Explorer, you can see this in action, the green highlighted processes are starting, red highlighted processes are ending:

Loose Threads, Unresponsive Edition

Staying responsive to the user is an important rule for software and to do this, the above-mentioned threads do ‘work’ in the background, leaving the user free to keep using the application, and most importantly, being able to cancel the operation in progress.

The Backblaze software ignores this key necessity, and as a result the windows consistently dim and then deemed unresponsive.

For example, when Backblaze does a full scan for new files, within seconds the window[2] stops responding and Windows will prompt you to forcibly close it. In the year I have used Backblaze, I never saw the progress bar get past the beginning:

In the Backblaze Downloader, once a download starts, any interaction causes the window to go unresponsive, making it impossible to see the speed of the transfer and making it impossible to cancel the download without fighting with the window (additional software used below to highlight mouse actions):

Backblaze Downloader goes unresponsive

The NeverEnding Polling

Windows supports multiple ways for processes to send information back and forth directly in memory (named pipes are a personal favourite), but despite Backblaze always running several processes, none of these communication methods appear used. Instead, Backblaze software reads several XML files containing the backup status from your hard drive constantly, even when no backup is in progress and the files have not changed. See below the various parts of the Backblaze Control Panel with their associated XML files and contents highlighted:

Using Process Monitor, lets observe this action, from bad to worse to ugly. First, when the backup is not running or paused, the Backblaze Service is parsing the same XML files over and over, every ten seconds:

Process Monitor showing Backblaze Service reloading XML every 10 seconds

Then it gets worse, the Backblaze Control Panel, when opened, reads the same XML files every one second:

Process Monitor showing Backblaze Control Panel reloading the same XML files every 1 second

Finally, while uploading to Backblaze, the re-reading of the XML files speeds up to virtually no delay (this is in addition to all other reads/writes needed to do the backup):

Process Monitor showing Backblaze processes going insane reloading XML files during backup process

The good news is that Windows will cache most of these reads, but if you are low on memory or doing intensive work, your performance will needlessly suffer from running the Backblaze software. Additionally, if you are on battery power, this infinite loop polling and parsing is going to lower your available battery life.

Restoring files

Although restore is a key marketing point for Backblaze, most notably being able to request a USB or hard drive with your data sent to you, the restore experience on the website is utterly frustrating at times due to its lack of standard file search/browsing features.

Search

Each time you visit the restore page of the website, there is a perfectly reasonable waiting period to load up all your files. However, typing in the search field and pressing the enter key does not perform a search, instead the restore page reloads from scratch, reloading all the files again and no search results are displayed. In order to perform a successful search, you need to use the mouse and click on the search button.

Browsing folders

To find a file based on a folder, you make use of a standard tree-hierarchy user interface. However, if a folder name is too long to display, the name is truncated with an appended ellipsis, and no way to show the full name. (The name does not appear while hovering or in the source code of the page.)

Additionally, there is no way to search for a folder, you are stuck to finding folders yourself by browsing.

Browsing folders on Backblaze being unable to see the name of the folder

Browsing files

Once you find the folder you want, browsing the files can be even more difficult. There is no way to sort by Name, or Size, or Date, and although sometimes the listing appears to be in alphabetical order, in some folders it seems completely random or the alphabetical order restarts half way through:

Browing files in nonsortable random order on Backblaze

Conclusion

Backblaze lost file pie chart

I uploaded nearly 6 TB in total from 6 individual internal hard drives onto Backblaze. When one drive failed, I attempted a full restore and found that not only was most of that drive not restorable, but Backblaze had lost 4.2 TB, or 71% of all my files.

Backblaze’s support team acknowledged the issue but could not provide a solution or explanation that fit the facts, blaming my failed drive for their software being unable to “self-heal”/re-upload my formerly intact files from all my six drives, ignoring my follow-up questions, and refusing to do any investigation.

Support proposed I request a refund or alternatively remove and restart my backup from the beginning. After I revealed that I had recovered the lost data from my own backup and sought to see their purported “self-healing” fix the problem, I was told that was not an option. My support request was closed automatically, and no further communication was received.

Given Backblaze’s insistence on their ability to reconstruct data, and “focus on ensuring data integrity”, my expectation is that any data loss of this magnitude would be met with full transparency, a team of engineers wanting to research and scrutinize every detail, daily updates, and a full report explaining what went wrong. Instead, left without any rational reason Backblaze failed catastrophically, or what they plan to do to stop it from happening again, any of their reliability claims seem dubious at best.

Backblaze’s software is not designed to operate the way Windows expects, thereby causing overall computer performance to suffer, the user interface to go unresponsive, and makes the entire service appear unprofessional. Additionally, restoring files is challenging by the lack of search and browsing features, and the website blissfully refuses to acknowledge when its unable to restore multiple files, leading users to think their files are safely backed up when they are not.

Based on my experience with Backblaze, they made it precisely clear how much they value a customer and their data, and that is the full value of a refund, exactly 50 US dollars, if you ask for it.

Lastly, as Backblaze refused to explain how they lost most of my files, I think I might have figured that out on my own.

Backblaze, self-heal thyselfBackblaze self-healing ambulance

In support requests and throughout this article, Backblaze cites “self-healing” repeatedly as an explanation. From my perspective, any “self-healing” should only be necessary in the extremely should-be rare situation when a single bit is damaged and unrepairable on the hard drive. Indeed, five years ago in a Reddit AMA, Backblaze says:

We wrote a “self-healing” functionality that checksums every single file on your system before it is ever uploaded. Then, our system constantly checks every file in our entire storage farm and makes sure that the file we have is exactly the file you had on your system. If it ever doesn’t match, we automatically reach back out to your system and upload that piece again.

For this reason, several times a year I validate the integrity of my files and did so immediately after Backblaze lost my data – there were no errors found. In addition, my Backblaze support request in December involved files re-uploading to the service that I verified were identical to those already on the service and fully recoverable. Support claimed this was normal for “self-healing”.

This “normal” response sounded questionable until last month, when Backblaze posted the following in another reddit thread:

Way more likely than a cosmic ray, sometimes the Backblaze technicians decide it is time to retire a pod or vault in the Backblaze datacenter because the drives in that pod or vault are too small and no longer a good financial decision to use them. The techs start the process by FIRST having that pod or vault stop accepting uploads. Next, the technicians issue a special command that asks all the clients that are still phoning home to please retransmit all files they have stored on that one pod or vault to somewhere else in the Backblaze datacenter. But realistically only 90% of the clients will correctly respond and do this, so the final step to decommissioning a pod or vault is the system copies the remaining 10% of the data off to a special archive area on ANOTHER vault.

They go on to explain that this re-upload process is an optimization to lower costs, by using the customer’s own computer to do the work of moving the data to the new location.

Furthermore, Backblaze’s subreddit has plenty of examples of this re-uploading of files with no resolution. From 2015, we see multiple people experiencing re-uploading occurring, which was supposedly ‘fixed’; from June 2017, “files uploaded many times,” the support people claim it’s “caused by the distance from data server”; the latest from November 2017, “my pc is re-uploading about 500GB of video files that haven’t changed,” with several more people offering confirmation that the re-uploading is happening to them too.

I postulate that sometime at the end of December, the aforementioned pod retirement/re-upload process was triggered on the bulk of my files. Unfortunately, a human, hardware or software problem occurred and as a result, Backblaze lost most of my data and possibility the data of the “10%” of customers whose software did not re-upload.

This theory explains why Backblaze considers “self-healing” to be normal, insists it was the cause of losing my files, and why no further investigation was deemed necessary.

The idea of distributing the workload of maintenance to the users is not a bad one, but it this design should be made perfectly clear when you sign up and should be an available option to opt-out of. If this truly was the cause of the loss of over 4 TB of my data, I should have been informed immediately, certainly better compensation should have been offered, and no further work using this method ought to be done until both the cause is established, and a provable solution implemented.

If you are a current Backblaze customer or prospective customer

My experience leads me to consider trusting data on Backblaze to be a game of Russian roulette and as such, you will need another service or local backup as you cannot trust your data will still be available on Backblaze when you need it. I would recommend regular downloads of your files to verify your files are undamaged. This needs to be done manually, as Backblaze will not give you any notification that any part of the restore has failed. You should also expect to be re-uploading your files regularly whenever they do retire a “pod” of hard drives.

Technical notes

  1. All data units are expressed in powers of 2, 1 TB = 1024 GB, 1 GB = 1024 MB.
  2. Backblaze has different scheduling options for when they upload files. The screen capture shows the manual or “Only when I click <Backup Now>” option. The default option is “Continuous” and does not show the unresponsive behaviour. However, I have experienced issues with the Continuous mode, multiple times support has suggested I turn it off, and their own tips (“Rescan Your Hard Drive to Check for Changes“) suggests to hold down the Alt key and press the Restore Options… button, which triggers this window and unresponsive behaviour to occur.
  3. Here is a per-drive breakdown of the 4.2 TB that was lost by Backblaze:
    Backblaze per-drive failure breakdown

 

Advertisements

Backblaze support transcripts

The following are the full transcripts of all my support communication with Backblaze technical/customer support. This post is intended as reference for my post Backblaze one year review: 70% data loss unexplained, mediocre software, distrust. All redacted information is identifiable by gray curly brackets, {}.

Backblaze support

While dealing with support teams, I do my best to be respectful, uplifting, and positive in an effort to offset the majority of negative requests they receive. That said, in my dealings with Backblaze, they were usually quick to respond but frequently provided inaccurate or incomplete answers. Not including the final support request involving my lost files, I would rate the support just average.

All dates/times listed are in the America/Toronto timezone.

Index

Request # Title Date begins
329150 Failed restore January 23, 2018
321691 Some files reuploading after recent event/beta client bug? December 29, 2017
320850 Backup restarted from beginning, now doing dedup slowly, any way to speed it up? December 26, 2017
297101 Folder that starts with three dots not backed up September 19, 2017
294635 Notification area icon not being restored bug/feature request September 9, 2017
272684 Missing files May 18, 2017
272393 Olark {Backblaze website} chat with {my e-mail address} May 17, 2017
266722 Frequently stuck producing file lists on pause/resume April 17, 2017

Request #329150:
Failed restore

January 23, 2018 00:59

Hi there.

I had another drive fail last night, and I’m trying to download my files from Backblaze.

I selected a major folder to restore, and it says:
Selected:
71,838.04 MB

But the restore is only 1.8GB, and in the zip archive most of the files are missing. I stepped back and selected some of the files that are missing manually, and the restore took only a few seconds to prepare and show up as “Failed”.

Is there anything I can do?

January 23, 2018 12:56

Hello,

Thanks for reaching out.

In order to see what’s going on with your backup we’ll need to get the password for your account. Feel free to change this to something you feel comfortable sharing by logging into your account at Backblaze.com, clicking the My Settings link in the left-hand column, and changing your password there.

Best,

{Support 1}

Get to know me! https://www.backblaze.com/blog/{Support 1 link}
The Backblaze Team

January 23, 2018 14:03

Thanks for responding so quickly {Support 1}.

I’ve changed my password for you, so here are my credentials now:
username: {Username}
password: {Password}
I’ve also turned off 2FA.

I’ve done more attempts at restores, and for the most part, no restores seem to be complete. In one case I selected a whole group of files at 75GB, and got back a 128KB zip with a bunch of text files. Trying to restore individual files from those folders results in a ‘Failed’.

As an example, if you attempt to restore T:\Lorax, which is an old video project of mine I did for one of my sister’s plays. The selected files are 8,344.64 MB, but the resulting zip is only 1.55GB, and attempting to restore the main file (also missing from the zip), T:\Lorax\Lorax Youtube 1080p.mp4, results in a Failed response.

Another example, T:\TapesEdit, which are my childhood cassettes I digitized and edited at one point. Selected: 7,409.11 MB, but the resulting zip is 463.7 MB. Attempting to restore one of the missing files in the zip like T:\TapesEdit\Flubber.wav, results in a Failed.

Also of note if it’s important, T: is the drive that’s failed on my side.

Let me know if there’s anything else you need. Thanks in advance for your help 🙂

January 23, 2018 19:33

Hello,

Thanks for your patience while I worked with our devs to look into this.

Please feel free to reset your password back and re-enable 2FV as we will no longer need access to your account.

After further review, it appears that several large files in your backup from the T drive are missing parts from their file. Typically when this happens Backblaze will go through a process called self-healing where the missing parts of a file will be reuploaded. However in your case it looks like 2 files from the 2 drive were never able to be backed up at all for some reason which kept the backup in an initial backup state. The self-healing only occurs once the initial backup is complete, so because your backup never left the initial backup state the self-healing never occurred to fix these files in your Backblaze backup.

Unfortunately because of this we do not have the several files from your T drive in your backup and there is not a way for us recover them as good versions of the files were never uploaded to our servers successfully. What appears when you download folders from the T drive now is the entirety of what is on our servers for the drive. This was a failure of the Backblaze system to work correctly with your computer and drives and if you did want to cancel your Backblaze service we would be happy to offer a full refund of your subscription.

If you would like to continue with us then a fresh backup would be necessary to make sure that files are all backed up correctly. If you need help with this I’m happy to walk you through this process to make sure there are no issues with your backup going forward.

Best Regards,

{Support 1}

Get to know me! https://www.backblaze.com/blog/{Support 1 link}
The Backblaze Team

January 24, 2018 01:32

Hi {Support 1}

Thanks to you and the rest of the team for investigating so far.

I did some further tests tonight, and the data loss is not limited to the T drive. For example, trying to recover a folder of 13.4GB from the D drive, results in a 43.08MB zip file.

In addition, I did manage to determine what’s left of T on Backblaze and I’ve found that only 423GB is still on the server, which means 1,103.12 GB has been lost from T (1 GB = 1024MB). I’ll go through the other drives later on to determine the exact number, depending on what happens next.

That said, I do have some follow-up questions, if you don’t mind.

To begin, you have said that this self-healing didn’t happen because of two stuck files. The thing is, I saw this self-healing process last month – see tickets #320850 and #321691 with {Support 3}. In the first ticket, it restarted the backup from zero and went through all the files. In the second ticket, {they said} that the self-healing and re-uploading I’m seeing is normal, even though the files were being re-uploaded that were already on the server (and recoverable with the same hash), however after that ticket was closed, files that were later being re-uploaded were not able to be recovered (and also had the same Failed state that I’m seeing now), but after they were re-uploaded and processed, those files were once again recoverable. Is that not the self-healing?

Back to the two stuck files. When I saw this in the UI, I tried to determine what those two files were and didn’t have much luck. Looking again through bztransmit23.log, I see “ATTEMPTING TO DECLARE VICTORY on initial upload” and then “BackupSummaryUserWouldSee=Selected_1,508,660_files_/_6,028,700_MB__Remaining_508_files_/_3,728_MB
NOT DECLARING VICTORY on initial upload – too much left”
However, the remaining files that are noted in the log never goes lower than 482, but the UI only shows two (and you also mentioned that there’s two). Looking further, the files are either noted as no longer existing, BZ_FILE_TEMPORARY_FILE_BUSY, or BZ_FILE_TEMPORARY_OTHER. There is also far more than two in every ‘diagnosed’ file, so I have no idea what the two files are. I have zipped/attached my full logs to this message and hopefully you can figure it out better than I.

My second question – is there any explanation for how this happened and why it won’t happen again? From my perspective, this plane fell out of the sky, half the people died, and the solution is to build another one and hope for the best? I already know that at least a terabyte is gone and I’ll gladly provide any info or data you need to better analyze and figure this out, but from this side, it seems like you guys don’t care to know or you do know and it’s some sort of secret.

However, I do have some good news for you. I have another backup of the T drive here, which was done the night before this all happened and I’ve restored it to a new disk tonight. With my data back, I would like to test whether this self-healing solution will work. Is there any way we can do this, or is there really no hope in doing so?

My apologies for the long message, I did my best to edit it down. Thanks in advance for your responses!

-Jon
Attached: {bzlogs-jonathankay.7z} (8 MB)

January 27, 2018 00:22

Hi Jon,

My apologies for the delayed response here.

Your description of self-healing is correct – large files from your backup on our servers are missing chunks of the file and Backblaze goes through the self-healing process to correct this and make files available once more via the Restore browser by attempting to reupload the missing chunks of the file(s) in question.

We can’t say exactly why this happened but it’s likely that either the drive had been malfunctioning in some capacity for some time that could have caused this. The self-healing is attempting to complete but has not been able to and will not be able to because of the drive failure.

I’m glad to hear you were able to recover the data off the T drive. The only way to test this would be to start a new backup which will rebuild the data files that keep track of the hashes of these chunks to prevent this from happening in the future.

Best Regards,

{Support 1}

Get to know me! https://www.backblaze.com/blog/{Support 1 link}
The Backblaze Team

This request is closed for comments. You can create a follow-up.

Status
Solved
Priority
Urgent

Request #321691:
Some files reuploading after recent event/beta client bug?

December 29, 2017 15:35

Hello my Backblaze friends. Earlier in the week I opened a case, ticket #320850. Quick summary of that: I had a process with a memory leak which somehow managed to get Backblaze to start again from the beginning. Although it was recovering, it was doing so very slowly, and {Support 3} instructed me to download the beta version of the client and I was back in business in under 24 hours.

However, just yesterday I had reason to pause the client briefly and afterwards it decided it needed to re-upload a large segment of files (maybe a good 100GB or so). This is not the end of the world but it will take a good while on my connection and is a bit of an annoyance. I restored a few of the files and checked the MD5/SHA-1 against my copies, and they appear to be identical, so I’m not sure why they need to be re-uploaded.

I’m reporting this problem in case there’s something you want me to try or if there’s a bug in this beta client that needs addressing. The compressed log files are well beyond the support forms 20MB limit now after my previous problem, so I’ll need to be more selective or upload elsewhere if you require the logs. However, I used your explainfile utility on one of the files and it showed a few interesting things, such as:
– line 8867 – 20171229044543 – ERROR – bz_done_ INCONSISTENCY_FOUND – 20171229044543 – DetectedDifferentFileId_InstructionMinus

I’ve attached the explainfile results.

Thanks!
Attached: {Backblaze Explainfile.txt} (20 KB)

December 29, 2017 21:15

Hi,

It appears that Backblaze has been performing some “self healing” which is when Backblaze checks what’s in the logs of what’s been transmitted in the past vs. what’s on disk vs. what’s on our servers. When inconsistencies are detected, it schedules the files for upload and then scrubs the logs of the inconsistent transfer.

Normally this process only picks up a few hundred out of the few hundred thousand files a typical backup has. However the amount of data experiencing this implies that a sizable number of files were detected as needing healing. Something about the initial backup of those files wasn’t consistent, so Backblaze is working through them one by one to correct them.

So there’s a few choices here:
You could allow the self healing to keep working away at these files and it may take up to 90 days to complete it in some circumstances
If these files are unimportant to be backed up, you could Exclude them from Backup, and Backblaze should ignore them after about a week or so
You could start a fresh backup, which will likely resolve the issue, but will involve reuploading all files

If you have any questions, please reply and let me know.

Cheers,

{Support 3}
Support Technician
The Backblaze Team

December 31, 2017 14:15

Thanks for responding again {Support 3}. So this is normal behavior? I admit I am a bit curious from a technical perspective what those inconsistences would be, since the hashes are the same of the files, although I am aware that files are more of an abstraction in Backblaze and files are ultimately stored as shards. I assume encryption makes it difficult to reasonably repair or resolve this other than reuploading? Although I am pleased to find the client airs on the side of caution, as there’s nothing worse than finding out a “working” backup has broken files.

With regard to the options, I don’t think I could practically exclude the files without going crazy, as they’re random on disk other than being a certain size. When I noticed it was starting with some 30MB files, and it’s up to 46MB files now. So in real terms that’s mostly small video clips, some FLAC music, a few podcasts I’ve archived, etc. which I definitely want to be stored correctly anyway.

All that said, I think I will let it do its thing for now and I’ll let you know if anything else changes. Thank you for all your hard work!

Take care 🙂

Jonathan

This request is closed for comments. You can create a follow-up.

Status
Solved
Priority
Urgent

Request #320850:
Backup restarted from beginning, now doing dedup slowly, any way to speed it up?

December 26, 2017 14:34

Hello Backblaze people! Happy holidays to you! Although if you’re reading this the holiday is most likely over, but have some chocolate on me 🙂

I arrived home late last night (December 25th) to find that an unrelated third-party process was experiencing a memory leak and was consuming most of the RAM+pagefile. So, I just rebooted the system normally.

After the reboot, I took a look at the Backblaze Control Panel and it indicated that nothing on the PC had been backed up and was starting again from the beginning! Knowing the basics of how Backblaze works, as I expected, it did start going through, scanning files and deduplicating, and that’s what is still doing now. However, this process seems pretty slow and appears to be using very little disk activity. I’ve seen this behaviour before when I lost a drive and it had to recover from that, and it took quite a while. The bztransmit log repeatably says “BzFile::ReadLargeFileIntoBuf_Slowly – called by LoadHashTableWithBzChunkFileIds, pausing for 4 seconds to lighten disk I/O”. Is this what is causing the slowdown?

Is there any trick to getting the Backblaze client to turn off the throttling, or other ways to make this process quicker? I did try increasing the number of threads with no improvement. I’ve attached a 7zip of all my logs in case it’s useful.
Attached: {bzlogs.7z} (20 MB)

December 26, 2017 23:34

Hi,

Thanks for reaching out.

I believe you may be correct in your assumption of what is causing the slowdown. This issue is actually addressed in the current beta release for Backblaze.

I would like to have you first open the Backblaze settings, navigate to the schedule tab, and then change the backup schedule to “Only When I Click”. Once you have changed the setting, close Backblaze, and reboot your system.

After the reboot, download and run the beta installer, available here: http://files.backblaze.com/

No need to uninstall the existing version of Backblaze, simply run the new installer in place. Once the installation is complete, reboot the system once more.

After this second reboot, open the Backblaze settings and change the backup schedule to “Continuous”. Once you have done so, please hold the Alt or Option key and click the “Restore Options…” button in the Backblaze Panel to force an immediate rescan on your machine. Let Backblaze run uninterrupted for at least 24 hours.

Please note that both restarts are vital steps in this process. Please do not skip the reboots in these steps.

Let me know if this doesn’t alleviate the problem.

Cheers,

{Support 3}
Support Technician
The Backblaze Team

December 27, 2017 13:08

Thank you {Support 3}, I have followed your instructions and it is performing significantly faster now. Thank you for all your help!

Jonathan

This request is closed for comments. You can create a follow-up.

Status
Solved
Priority

Request #297101:
Folder that starts with three dots not backed up

September 19, 2017 11:31

Hey backblaze people!

I noticed a few days ago that one of my folders wasn’t on Backblaze, when it technically should’ve been uploaded a long time ago. The folder in question contains three ~7MB MP3 files from an album called “…To The Music” and the folder (truncated this a bit for readability) is D:\Music\…To The Music\.

My current guess is that this is being ignored as I believe the uploader ignores folders that start with dots (like your .bzvol folders). Is that correct? I did make an effort to find documentation on whether what I’m seeing is intended or not, but couldn’t find anything definitive.

Also, is there any way to fix this, other than renaming the folder? I took a look at the exclusions xml but didn’t see anything in there that would cover dotted folders.

Thanks in advance for any advice or suggestions! 🙂

September 19, 2017 16:23

Hi;

Thank you for reaching out to us today.

Backblaze has a diagnostic tool called “Explainfile” which can help determine why these files aren’t backed up accurately.

I’ve attached a script to run the diagnostic. Unzip the script to a convenient location, and drag and drop an example file that’s not backed up correctly on to the script. It will output a Backblaze Explainfile.txt to your desktop if it was run correctly.
Note: you can not double click the script to run it, you must drag and drop a file to it.

Please submit the outputted Backblaze Explainfile.txt to your response.

Regards,

{Support 2}
Find out more about me: https://www.backblaze.com/blog/{Support 2 link}
The Backblaze Team
Attached: {Backblaze Explainfile Win.zip} (478 Bytes)

September 19, 2017 22:17

Thanks for all your help {Support 2}. I’ve done that and the results of each file were PRIMARY_DIAGNOSIS: healthy_and_found_in_bz_done.

However, I went back into the View/Restore Files and noticed this magical option, ‘show hidden files’, and sure enough, there was the folder and the MP3 files.

My sincere apologies as I had actually forgotten about this important detail — this folder was in fact set as hidden when I noticed this now-not-so-missing folder on Friday and I “unhid” it at that point. I had checked the bztransmit logs to see if it transferred after removing the hidden attribute as I assumed it was never uploaded, but obviously it had no reason to upload since it was already there.

My follow-up question for you is since the files and folder are there, will my removal of the hidden attribute change on the folder eventually trickle down into my backup or is that not something the uploader covers?

I’ve still attached the file in case it’s helpful 😊

Also, as I write this, {relatable personal information about support 2 from their Backblaze blog entry}, so you’re in good company!

Thanks so much!

Jon

September 20, 2017 16:14

Hi Jon;

I am glad to hear the files were backed up!

If you have changed the folder from being hidden to visible, that should update in the next couple of backups. It may take 48 hours for the change to be reflected in the backup.

I would recommend checking the backup in about 2 days and see if checking the box to not show hidden files/folders makes that folder disappear again.

If the folder is still hidden, let me know.

Best Regards,

{Support 2}
Find out more about me: https://www.backblaze.com/blog/{Support 2 link}
The Backblaze Team

This request is closed for comments. You can create a follow-up.

Status
Solved
Priority
Urgent

As of March 2018, this folder is still shown as hidden.

Request #294635:
Notification area icon not being restored bug/feature request

September 09, 2017 00:46

Greetings Backblaze!

An issue that has been bugging me for quite some time is when Windows Explorer (explorer.exe, ie. the taskbar) crashes or is otherwise restarted in some way, the Backblaze Control Panel icon does restore itself into the notification area/system tray. I admit I was slightly disappointed it wasn’t resolved with the latest release.

I understand that the team is concentrated on the more functional aspects of Backblaze, but handling this simple situation really is expected for any application that runs in the notification area, and the required API/window message to make this happen has existed since IE4/Windows 98, which is why I label this first as a bug.

Adding this function is not complicated and would take just a few minutes, and is well documented in MSDN: https://msdn.microsoft.com/en-us/library/windows/desktop/cc144179.aspx#Taskbar_Creation_Not

One just has to register for the window message and then run your Shell_NotifyIcon initialization again when the message comes.

In terms of repro steps:
1. Have Backblaze running
2. Open up Task Manager
3. Click on Windows Explorer
4. Choose Restart button
5. Note how every other application in the notification area comes back except Backblaze

Alternatively, you can kill the explorer.exe process in Task Manager, and then start it up again, or on Win10, you can hold down Ctrl-Shift, right-click on an empty area of the taskbar and choose Exit Explorer.

Keep up the good work. Thanks so much!

September 09, 2017 14:41

Hello,
Thanks so much for the feedback & detailed info.

I will pass this info along as desired.

1. Does the icon re-appear after about an hour or so?
2. How do you restore the icon after reproducing the behavior?
3. Does restarting also restore the icon?

Please let us know, so we may investigate further.
Thanks,

{Support 4}
Tech. Support Tech.
The Backblaze Team

September 10, 2017 16:35

Update:
Our dev took a look at it, and was able to squish the bug. He said to tell you ” thanks and it should appear in a build soon. ”

No need to supply the additional info, thanks again.
Sincerely,

{Support 4}
Tech. Support Tech.
The Backblaze Team

September 11, 2017 10:18

Wow, thanks a lot {Support 4} and everyone else involved. I am very much impressed. I look forward to that in later releases and I think others will benefit from having that sorted out also.

Now about my other feature requests… just kidding 😉
Take care!

This request is closed for comments. You can create a follow-up.

Status
Solved
Priority

Request #272684:
Missing files

May 18, 2017 23:43

Hi there. I scanned some pictures earlier in the week, placed them in their own folder and went about my business. According to the C:\ProgramData\Backblaze\bzdata\bzlogs\bzreports_lastfilestransmitted\16.log file, the files were transmitted to the server in the evening on the 16th but although other files (in this case, other scanned images in other folders I did later on) in the log file show up in my account, the one particular folder and its jpg files do not appear to restore. The files also don’t appear if you search.

It’s my understanding that the files should show up within 24 hours.

The missing folder is “D:\Users\TReKiE\Pictures\Scans\High School\DSC 4AI CompSci\An Intro to C” and the files inside should be book 001.jpg, book 002.jpg, etc.

May 19, 2017 15:55

Hello,

Thanks for reaching out!

It’s possible that these files may not have uploaded yet as large files (over 30 MB) only upload once every 48 hours to conserve bandwidth. Please wait at least 48 hours from when the files were added to your computer and check again on your account online.

Best Regards,

{Support 1}

Get to know me! https://www.backblaze.com/blog/{Support 1 link}
The Backblaze Team

May 20, 2017 15:03

Thank you {Support 1}, always glad to hear from you!

I wanted to let you know that I went and checked again this morning at around 09:45 PDT and sure enough, all the missing files are now there. I did check last night too, according to browser history, 22:23 and the files weren’t showing up at that time.

The original batch of files was transmitted (per \bzlogs\bzreports_lastfilestransmitted\16.log) on the 16th at ~17:00 PDT, so some wolfram math makes that about 3.5 days or about 84 hours. The files themselves are just JPGs at 600KB to 1.5MB, but there was ~200 of them. There were some larger files uploaded at the same time and since that were >300MB and those all showed up fine within hours, it was just these images that were missing.

I only tell you this stuff in such detail because I used to have your job, metaphorically speaking, about 10 years ago (just at Streamload/MediaMax, as far as I know, the first “unlimited” storage provider) and so I know how difficult it is to get reliable user feedback on how the system works and how important that information is doing good CS/tech support.

Have a great day!

Jonathan

May 21, 2017 01:36

Hey Jonathan,

Thanks so much for the detailed feedback!

I’ll be passing this along to our devs to make sure we address any bugs present here.

All the best,

{Support 1}

Get to know me! https://www.backblaze.com/blog/{Support 1 link}
The Backblaze Team

This request is closed for comments. You can create a follow-up.

Status
Solved
Priority
Urgent

Request #272393:
Olark {Backblaze website} chat with {my e-mail address}

May 17, 2017 17:30

Location: Canada (Waterloo)
Referred from: https://www.google.ca/
On page: https://www.backblaze.com/help.html
IP address: 99.243.12.29
Talked with: {Support 1}

{Support 1}
03:26 PM | Hi Jonathan, how can I help you today?

Jonathan ({my e-mail address})
03:27 PM | Hi {Support 1}! just a quick question i think, i was watching the uploader yesterday upload some files and they don’t yet appear to restore yet, i do believe that’s normal based on my observations previous but i’m curious how long it should be, just generally, for files to show up.
03:28 PM | appear in the restore file listings i mean

{Support 1}
03:29 PM | Generally it will take about 2-4 hours before a completed upload is available for Restore online, but it can take up to 24 hours before they show as available.

Jonathan ({my e-mail address})
03:29 PM | alrighty, thanks for the info, most appreciated 😀

{Support 1}
03:30 PM | No problem!

Jonathan ({my e-mail address})
03:30 PM | Have a great day 🙂

This request is closed for comments. You can create a follow-up.

Status
Solved
Priority

Request #266722:
Frequently stuck producing file lists on pause/resume

April 17, 2017 23:52

When I press the Pause Backup button and then choose Backup Now again, 8 out of 10 times, it will get stuck with “Producing file lists…”, CPU usage on the backblaze bztransmit64.exe is virtually 0 and using process monitor, i can see that it is accessing files, but at an extremely slow rate. To fix the problem, I restart the Backblaze Service, hit Backup Now again, and it now works as expected. In fact this happens so often, I always restart the service prior to resuming it, to make sure it works. This is pretty annoying though, is there anything I can do assist in getting this issue fixed?

Note that I am in the Initial Backup phase.

April 18, 2017 17:58

Hi;

Thanks for writing in!

If you are needing to pause the backup for some reason, change the schedule to Only when click. That will stop the backup. Then when you click “Backup Now” it will start backing up again and you should not see the same issues.

When you click on Backup Now, even under continuously after pausing it, it can take some time for the backup to resume as it needs to scan the machine again to make sure it restarts from the correct spot.

Please let me know if you still encounter any issues.

Regards,

{Support 2}
Find out more about me: https://www.backblaze.com/blog/{Support 2 link}
The Backblaze Team

April 21, 2017 14:59

Hi {Support 2}

I tried this over the past few days to see if it would fix the problem. Although it did seem to be initially successful, last night I paused the backup to send an important large email quickly, and then pressed Backup Now when complete. I then locked the computer and let it do its thing. I had a busy morning/afternoon and now that I check, 13 hours later, Backblaze says “Producing File Lists…” and does not appear to be doing anything. The window indicating that Backblaze is scanning the disk had finished, but never went beyond that.

In other words, I still have the same problem despite the schedule change.

Going over to the Task Manager, Details tab, I saw that there are only two backblaze processes running. bzbui.exe and bzserv.exe. The bztransmit64.exe, which I expect to be there during the Producing File Lists phase of backup, was not running.

I pressed on Pause Backup again to see if I could get it to pause to try and restart it. Although the button selected fine (the bzbui isn’t frozen or anything), it did not pause the backup. So I went back to my original workaround, restarting the Backblaze Service, which then makes the Pause Backup button change back to the Backup Now button, and pressed the button. It then scanned the disk, processed the file lists in under 60 seconds and started sending out the backup again.

Upon checking the Event Log and Problem Reports, bztransmit64 never crashed, in fact no Backblaze process has crashed, so certainly that isn’t the cause of the problem.

I have a relatively normal configuration, a few drives, all NTFS, all single partitioned (other than the Windows drive with its EFI and Recovery partitions). No antivirus or weird security software installed. Backblaze set to automatic throttling. Are there any log files or similar we could be looking at?

Thanks for all your help so far.

Jonathan

April 21, 2017 17:15

Hi Jonathan,
{Support 2} is out right now but I will try to assist in {their} stead.

At this time, I’d recommend setting it back to “Continuously” to ensure it runs as often as possible. For even more info on the different Schedule settings and their functionality, we have this article: https://help.backblaze.com/hc/en-us/articles/217666488-Schedule-Settings-Win-
Then, to ensure the program is “healthy” we can try the following update procedure, which addresses a few different potential issues:
“- Please try disabling any network monitoring programs, firewalls and antivirus on your machine, then follow these steps to repair the program:

1. Go to https://secure.backblaze.com/update.htm
2. Download the installer for your computer.
3. Restart the computer. DO NOT skip this step.
4. When the computer is up and running, open the installer and click install now.

– Once you have done so, please try holding the Alt or Option key and click the Restore Options button in the Backblaze Panel to force an immediate rescan on your machine.

Let Backblaze run uninterrupted for at least 24 hours, and let us know if the issue persists.”

Then, also please keep in mind, the program cycles in it’s tasks every 1-3 hours, longer depending on a few factors, such as machine resources and amount of data (number of files AND amount of data).
Additionally, Computer & Drive Sleep settings will still Backblaze so that it wont run during these periods. For other recommendations we have this article: https://help.backblaze.com/hc/en-us/articles/217664608-Best-Practices

Please let us know if there’s any other questions or issues at all. Screenshots appreciated (One of our favorite screenshot tools here: https://www.take-a-screenshot.org/ ).
Thanks,

{Support 4}
Tech. Support Tech.
The Backblaze Team

April 24, 2017 03:14

Thank you {Support 4}

I’ve followed your instructions below and I’m afraid there’s been no change. I paused the backup earlier to upload a video to YouTube and after the 30 or so minutes it took to upload the video, I pressed Backup Now and it stuck at “Processing File Lists” with only the two Backblaze processes running – bzbui.exe and bzserv.exe. After restarting the Backblaze service in services.msc, I press Backup Now again and it proceeds as per normal.

My machine is set to the High Performance power setting in Windows and my drives are set to the highest performance setting in their Advanced Power Management (where available). There is no RAID and no external disks.

I’m not really sure what you would like me to screenshot, there isn’t a whole lot to see beyond “Producing File Lists…”. Please let me know if I can provide any further information that may help!

Jonathan

April 25, 2017 18:00

Hi Jonathan;

I would like you to try temporarily disabling your anti-virus and firewall programs

This is typically caused by a security program interfering with Backblaze.
These help article may explain how to do this:
https://help.backblaze.com/hc/en-us/articles/217664708-How-do-I-disable-my-firewall-

http://www.windowscentral.com/how-permanently-disable-windows-defender-windows-10

https://support.microsoft.com/en-us/instantanswers/c9955ad9-1239-4cb2-988c-982f851617ed/turn-windows-firewall-on-or-off

Then try pausing and restarting the backup after a few minutes. If you do not have the issue, it means those programs were stopping Backblaze from working. You will just need to add Backblaze to the approved/exceptions lists.

If you still have the issues, then I would like to collect some logs. I would like to gather some logs to into this further.

Log files are located at:
C:\ProgramData\Backblaze\bzdata\bzlogs\

Please click on this link for how to show hidden files: http://windows.microsoft.com/en-us/windows/show-hidden-files.

Please zip the bzlogs and bzdatacenter folders and attach them to your reply. The logs will contain file names and locations, but no actual files.

Regards,

{Support 2}
Find out more about me: https://www.backblaze.com/blog/{Support 2 link}
The Backblaze Team

This request is closed for comments. You can create a follow-up.

Status
Solved
Priority
Urgent

Note: There is no missing messages past this point as I did not respond to this message. I didn’t choose to invest any more time because I already had a workaround of my own (restarting the service), and they were at the point of blaming the Windows Defender and Windows Firewall, which made no sense given the technical details. Almost a year later (March 2018), this specific problem has become more rare, but still can occur.

AIM and the importance of preserving conversations

 

trillian_2017-12-15_22-17-42

Hm, not reconnecting…

I noticed around 1:30 AM EST today, December 15, 2017, that AOL Instant Messenger had disconnected for the last time, apparently right on schedule.

 

I have been signed into AIM since it became available to non-AOL subscribers in the late 90s, but really, I preferred ICQ, and only started to use AIM more regularly as ICQ lost popularity.  Then as Messenger began to surge in use, AIM became a secondary client with those few contacts who still used insisted on using it.  In more modern times, I delegated both ICQ and AIM to multi-service software, at this moment, that is Trillian.

Unlike Messenger, I never considered AIM itself that important, it was just a tool I used to communicate to others.  Although I’m particularly fond of Emily is Away and its sequel, I felt the software was fairly generic and didn’t offer that much.

When the announcement arrived of AIM’s shut down this past October, I went through my AIM contacts, reading through the screen names of friends that have long since added me on other messaging platforms, plus the endless sea of friends of friends with whom I hadn’t spoken to in over a decade.  How would I preserve this list of people?  Then I realized I already had, and a lot more.

Originally using AIL and then DeadAIM, any AIM conversations from the year 2001 onward were automatically saved, lovingly preserved in HTML, complete with “Direct Connect” assets, as well as the original AIM colouring,  and other formatting fully intact.  This archive comprises 8446 files and takes up 345MB of space, hardly anything for what I consider priceless data.

 

deadaimscreenshot

December 22, 2004 – AOL used to send out cookies for Christmas

Unlike other communication mediums, even other electronic ones, instant messaging has a huge advantage of being easily saved and preserved.  What I have found worrisome of late is that the more modern instant messaging tools are mobile-centric and lack easy logging capability and even worse, have limited options to allow third-party software to do this job for you (aka DeadAIM or Messenger Plus!).

In doing some research for this post, I was pleased to discover that work has been done to try and mitigate this problem (note: I have not tried most of these), for example, tg-export (Telegram), DiscordChatExporter, and WhatsApp Export.  Additionally, Google Takeout or Facebook’s “Download my data” will provide archive file(s) of their respective chat services.  However, despite being recently updated, Microsoft’s Privacy Dashboard still has no way to export your Skype conversations.  A significant problem if you’ve been upgraded to the non-Classic versions of Skype, or are using the built-in Windows 10 application, as they have no local conversation store.  Still, with all of these options, they require manual, regular updates to keep your archive current, as well as some degree of trust if the conversations are stored on the server.

After reflecting on AIM’s shutdown today, it seems that when instant messaging services shut down, it’s not about the software, it’s really about the communication and the memories you made while talking to others.  When you have those memories preserved, you have the ability to re-visit them whenever you want, regardless if the service is still running or not.

So, the next time you use any instant messaging tool, be sure to consider if you have a way to preserve those conversations, because no service will last forever but your archived instant messaging logs will.

A final note for those interested, my Messenger Plus! logs currently stand at 37,165 files/1.36GB.

The New Skype

Exactly two weeks since the last Messenger server twinkled out of existence comes the announcement of “New Skype“.  See Skype’s promotional video below:
At present, this is only an announcement and the software has not yet been released.  However from what is shown in the video, I have a few questions, observations and opinions already.Are there any (MSN or Windows Live) Messenger features?I generally evaluate new messaging clients by comparing them against Messenger’s standard set of features.  Based on what we know so far, I don’t see any sign of New Skype of backgrounds, custom emoticons, custom fonts, nudge, status (presence), or voiceclips.

New Skype Conversation

Background colours with gradients in New Skype

It does appear you can set a background colour (and gradient!) for the text of an individual message reminiscent of the background colour functions in the modern versions of Messenger Plus!.  However, it’s not entirely clear if this feature expands to allowing you to change the colour of the actual text.  In the screenshots, a background colour swapped the font colour from black to white.

 

Winks (called Stickers) do exist in New Skype and are animated, as well as the normal expected GIF support and Skype’s own “Mojis”.New Skype appears to have a Snapchat stories-like Highlights feature for material you want to share with all your contacts for a seven-day period.  Of course, this stories feature is the latest variant of what was once Messenger 2009’s “What’s new?” pane as well as 2011/2012’s social pane.Will all the problems be fixed?

Although I haven’t had a lot of exposure time to the iOS version of Skype, I have had quite a lot of time with the Android version and its whole slew of reliability and performance problems.  From time to time, I’ve resorted to uninstalling Skype on my phone because of weird glitchy behaviour that has made the phone unusable or drained the battery.  I do hope a lot of these issues have been ironed out since they’ve taken steps to drastically change the client.  Additionally, Skype recently ranked as using the most mobile data in comparison to other mobile messaging clients, and hopefully that has also been improved.

Why so much white?

Everyone I have shown the promo video to has had the same immediate reaction I did – there’s just way too much white space.

Old vs New Skype

Old Skype vs New Skype

It reminds me of Office 2013’s attempt at white everywhere in its default theme, which was also unpleasant

 

Particularly on phones, I can’t think of anyone who wants to be blinded by a mobile screen first thing in the morning or replying to a message late at night in the dark.

Although I would prefer a dark background option, I rather like the look of the current mobile Skype.  It retains the classic Skype colour and fits in with whatever native operating system it’s on.

Will it be compatible with current Skype?

I would assume yes, but it’s not explicitly mentioned anywhere.  I find it a bit weird that I would have to consider such a question, but Yahoo! had no qualms in completely eliminating its existing Yahoo! Messenger service/contact lists last year in favour of the “new Yahoo! Messenger”.

What about the desktop version?

As part of the announcement is Windows and Mac versions.  I do wonder if the Windows version will be yet another variant of Skype.  At present, there are two Skype Windows versions – the classic Win32 version you download from the website and the new Universal Windows version (excluding Skype for Business versions as that operates on local servers or Office 365).  At the very least, I would assume New Skype to also be written for the Universal Windows Platform (UWP).

I use the Win32 version of Skype, primarily because it does have a (albeit limited) API that ties in with my headset’s software, notifying the headset base of calls, allowing a button on the headset to answer calls and an LED to let others around me know I’m in a call with someone. On top of that, the Win32 version provides a local message history spanning all the way back to my first main use of Skype, I can log into multiple accounts at once and particularly important, my taskbar icon for Skype lights up when I get a message (a feature not available in Universal Windows applications at present).

I don’t think any of the features in New Skype would be compelling enough to lose such valuable features from the classic version of Skype.

Final thoughts

Overall, this update appears to be an attempt to “keep up” with the other modern messaging services today and less about trying anything particularly new or interesting.  When the New Skype does get released in some form, I’ll do another comparison.

For now though my advice to the Skype team is simple, please, please, please provide a dark theme option.

Messenger Reviver 2 Discontinued

It is with a heavy heart that I announce that Messenger Reviver 2 has come to an end. reviver2.png

On May 18th, 2017 Microsoft shut down the last Messenger server, preventing Windows Live Messenger from being revived any further.

MSN Messenger first came online July 22, 1999 making public access to the servers available for a total of 17 years, 9 months and 27 days.  In the 2000s, Messenger dominated the instant messaging world, being used for text, voice, video, remote assistance, game invitations, and enjoyed both for business and personal use.  Massive communities and long-lasting relationships were formed around all around the world on Messenger-related websites and tools like mess.be and Messenger Plus!  Even today, despite hundreds of new messaging applications, still no client has yet to match the features and customization options of Windows Live Messenger.

Jonathan and Kurt

Messenger MVPs – Jonathan and Kurt Robaer at Microsoft, 2007

I originally got involved with the Messenger community back in 2001 as part of the Windows XP beta “Associate Experts” program, and quickly became a Microsoft MVP for Messenger, a position I held for over 10 years.  After Messenger officially “shut down”, I released Reviver to continue making Messenger connect to the Microsoft servers, and that has continued for over four years, with 98 versions of changes and fixes.  The original versions of Reviver only required a small adjustment to Messenger itself, but later on DNS servers, proxy servers and even modifications to Windows itself were needed to keep Messenger working.  I also expanded into versions on OS X/MacOS for Messenger:Mac and Adium, as well as aMSN.  It’s been quite a ride!

Previously, I have not taken donations for my work but as this is the end of the Messenger Reviver 2 chapter, I feel comfortable now putting out my PayPal tip jar for any who feels Reviver has been useful to them over the years.paypalme
Back in 2015 when it first appeared that the Messenger servers might not be available for much longer, I recorded a video of myself going through all the available versions of Messenger discussing and comparing the various features as well as telling some stories about the software.  I present it to you below or you can watch it on YouTube:

Although you can no longer use Messenger with the public Microsoft servers, you can continue using the older versions of Messenger on a new private server called Escargot.  You can find contacts to add on this forum topic.

This blog, forum and chat will continue operating, so please stay subscribed for news and discussion on Escargot and other Messenger related topics.

You have a several options to continue to communicate to your contacts on what was formerly known as the .NET Messenger network.  Of course, you can use Skype, but also Miranda-NG, Pidgin with EionRobb’s, SkypeWeb plugin, or SkypeWeb with other libpurple-compatible clients like Bitlbee.  You can also use the web-versions of Skype at https://web.skype.com or on https://outlook.com.

MessengerReviver_2017-05-19_22-41-38A final version of Messenger Reviver 2 is now available that you can download as a memento which retains all the original features you can explore or even just open it up to play around with the Easter eggs.

If you have looked at the Reviver About screen previously, you would have seen many important people who made Reviver possible when it was first released.  For the final release, I’ve also added the people who have been vital over the years I have been supporting and working on Reviver, and I would like to thank them here too: Emil, John, Tasos, Dean, Petri, Jeff, Mariano, Esteban, Peri, Alexis, dx, Jacko, Javier, and Raul, thank you for your friendship, endless discussions about Messenger, many late nights and supporting me and my projects.

Patchou and I

Jonathan and Patchou in 2006

Additionally, I would like to thank Patchou for creating Messenger Plus! Plus! was the cornerstone of the Messenger community for many years, indispensable for adding missing vital Messenger features and responsible for many relationships and careers for so many people.  Furthermore, I’d like to thank wtbw for always being available for reverse engineering assistance and pointers.

Thanks to Valtron for building the new private Messenger server and giving Messenger continued life.

Finally, a special thanks to all of you who have been users of Messenger Reviver 2 for so long.  You have made this journey a great and memorable part of my life and I greatly appreciate all the kind words and support over the years. Please stay in touch!  You can reach me on the new Escargot server by adding the user trekie@jonathankay.com, on Skype by adding trekie, my website, the forum, or here on this blog.

I think it’s appropriate to end with my favourite wink, the UFO, which can be both a hello and a goodbye.

signature
Jonathan Kay

butterfly-and-i-smaller

Hello there, MSN butterfly!

Messenger presently offline

At approximately 15:45 UTC, the Messenger bn2 servers have stopped responding.  Investigations are still underway but you will be unable to connect.

You will receive the following possible error codes: 80072efd, 81000306 and 80072efd 81000301.

If you are still signed in, do not sign out or you will not be able to sign in again at the moment.

EDIT: Scanning through IP ranges hasn’t found a suitable replacement, Butterfly Messenger and the Windows 8.0 client do not appear to be working either.  Skype is still working.

EDIT 2: Also of note, if Messenger cannot be restored, all focus will shift towards the Escargot server and continuing its development. Visit the forum for discussion and contacts to add.

EDIT 3: After more scanning and hoping, there’s no sign of the public Messenger server.  Unfortunately this means, Messenger Reviver 2 support will have to end.  If you haven’t already done so, please read my final Reviver 2 post as I’ve tried my best to pay tribute to a great messaging client and service.

Happy fourth anniversary Messenger Reviver 2!

What a dizzying four years, but Messenger is still (mostly) usable. Unfortunately more features such as groups (with a workaround) have stopped working, and conversion to Office 365 on Hotmail has caused a multitude of contact list issues, even for Skype-based Messenger users.

In case you haven’t been watching the forum, recently the ever awesome valtron has put up a Messenger Protocol server for older versions of Messenger as the Microsoft server stopped accepting older clients back in 2015. Give it a try!

Happy Anniversary! Birthday_cake

Messenger and the Windows Essentials installers removed from Microsoft servers

At what is no surprise to anyone, the Windows Essentials have disappeared from the Microsoft servers.

I did prepare for this situation for Reviver use and the Essentials installer files have been saved.  However, this does mean that installing Messenger for the time being will now be a larger download at 100MB.

Additionally, not all the languages will be available for the next few hours, if there is one you are in need of right now, please let me know in the comments and I’ll move it up in queue. All languages are now available. 

Messenger and Microsoft accounts down for some users today (error code 81000395)

This afternoon Microsoft account based services like Messenger, Skype, Xbox, Outlook.com and others are experiencing intermittent issues. 

In Messenger you’ll get error code 81000395 when you try to sign in. Additionally when you do sign in, contact lists may be incomplete.

Just keep trying and it should eventually allow you to sign in. 

You can also check the Skype heartbeat and Office.com/Office 365 status pages. 

Too many messaging systems?

In the heyday of Messenger’s popularity, I had very little reason to use any of the other instant messaging services on a regular basis, because (almost) everyone used Messenger!

However, today’s XKCD comic expresses my current situation perfectly:

chat_systems

To further explain, here’s a screenshot of the messaging section of my taskbar.
2017-03-13_15-41-02

There’s at least nine applications for instant messaging, not counting that some connect to more than one chat service.  Within each application, I tend to have a completely different group of contacts, and some are used more so for audio, group, or exclusively text communication.

Although over time I’ve become more accustomed to knowing which contact is trying to talk to me on which application, and the required CPU and memory usage for each application hasn’t been too noticeable in general usage, searching for information can be quite difficult as the message history for each contact usually can only be searched within the application or service itself (or virtually not at all, Google Hangouts).

Recently Microsoft has shown that they are working on a feature (called My People) for Windows 10 that would pin contacts to the taskbar instead of individual applications.  The feature has been delayed and may not work with existing desktop applications (which should be the subject of another article), so this may or may not help the situation.

Messenger on its own originally supported both the .NET Messenger Service/Windows Live and AOL Instant Messenger, and you can still see the basis of this if you hover over a contact in Messenger 2012 (look for the Networks: heading).

Additionally, Messenger originally supported a full API back in the 90s to allow others applications to fully communicate through Messenger without evening opening the program.  This was quickly abused and scaled back slightly but the API is still usable and was made use of in tools like Messenger Plus!, Outlook, Outlook Express and others.  Almost no instant messaging applications support this level of control now.

I’m curious how many messaging applications others use, so I’ve made a poll on the forum where you can indicate which applications you actively use on your PC.