Atlassian is still scrambling to recover from a recent software script fiasco and is hoping no customer data gets lost, which may be more than Microsoft can manage if OneDrive, as some have reported, has been intermittently corrupting large uploads for at least two months.
Four days after some Atlassian customers began encountering problems with the cloud giant’s collaboration software, recovery efforts continue and a few folks are worried they may not get their data back.
One wrote to The Register wondering about that possibility after the company, via Twitter, responded to a request to confirm that customer data is backed up and failed to actually do so.
“We expect most site recoveries to occur with minimal or no data loss,” the biz said on Thursday.
Minimal data loss of course is not the same thing as no data loss, and this, to those affected, is understandably troubling.
“This is extremely concerning to us, as our mission-critical institutional knowledge lives in Confluence at this point,” said one Atlassian customer, who asked not to be identified, in an email to The Register. “And that this message runs counter to the ‘maintenance script has disabled a small number of sites’ message we’ve been getting over and over again. This would also explain why it has taken days with so many engineers ‘working 24/7.'”
Atlassian Jira, Confluence outage persists two days on
The Register inquired to Atlassian about the IT breakdown, which from the company’s status page still looks to be an ongoing affair. Toward the end of the day on Thursday, Atlassian’s spokesperson provided a reassuring but similar statement.
“We are continuing to investigate and resolve the incident,” the spokesperson said. “At this point in time, we believe that any potential data loss will be minimal to none. We are working hard to resolve the incident and get customers back online.”
We were also told that the incident affects a relatively small number of Atlassian customers: about 400. That’s only 0.18 per cent of the company’s 226,000 customers, which isn’t much consolation to the several hundred who still can’t access their data.
When we checked back with our reader on Friday, the issue had yet to be resolved.
“No data has been restored to us yet, and these services all remain down,” our source said. “We have been given no ETA on access.”
When service has been fully restored, perhaps Atlassian will provide a detailed report on what happened using its Incident Postmortem Template.
Meanwhile, Microsoft’s OneDrive has apparently been corrupting large, multipart file uploads, intermittently, for at least two months.
Reports flagging the issue date back to early February, where they surfaced in a forum post for backup app Duplicati. Users of another backup app, rclone, started discussing what appears to be the same issue back on March 24, 2022.
Three days ago, Nick Craig-Wood, creator of rclone, posted a bug report to the GitHub repo for Microsoft’s OneDrive. “Sometimes (maybe one time in 20) multipart uploads of a 128MiB file get corrupted,” his post explains.
Multiple other individuals say they have reproduced this occasional bug.
“During testing this morning, I still see the problem,” wrote GitHub user “rleeden.” “I created a random test hierarchy of 100 files containing random data between 128M and 256M, and uploaded this to OneDrive via the web interface.”
“Checking the sha1sum of the original files and the files on OneDrive, I see that 16 out of the 100 files have been corrupted.”
Then on Thursday, this individual was unable to reproduce the error, suggesting someone at Microsoft may have fixed the bug.” And Craig-Wood also reported no longer seeing corrupt files.
The Register asked Microsoft, which was programmatically alerted to the bug report when it was filed, whether anyone at the company did repair OneDrive and, if so, whether any announcement might be forthcoming to OneDrive users who may not realize some of their uploads may be corrupted.
We’ve not heard back. ®
Updated to add
Microsoft got back to us simply to say the OneDrive issue has apparently been fixed.