A Showdown with Ransomware

In my career I’ve had encounters with Ransomware a few times. A few months ago we were attacked again. And rather than just recover from the effects, I decided to do some more investigation as to how the virus got into our network. The strain we were faced with this time is the ODIN Locky Ransomware.

First we were notified by one of our end users that several folders on our shared drive have missing files that were replaced with a strange looking file with the .odin extension. I was not familiar with the ODIN extension but as soon as I saw the files, I knew exactly what they were. Ransomware.

image

You may notice the first file listed is an html file. This is the ransom note. Typically it would also be displayed on the users desktop wallpaper but since this is on a network drive, we are unsure who the exact user that triggered the virus. The ransom HOWDO_ HTML looks like this:

image

By investigating the security permissions on the folders and files, you should be able to figure out who the end user is that triggered the virus. However, in our case, there was a security group that had access with a large member base so it would take forever to track down who caused it.

I did a little research on the ODIN extension and found some great articles

Information on the Ransomware variants that caused this:

Steps Taken to recover:

  • Deleted all affected folders/files
  • If you have identified the computer that spawned the ransomeware, then it’d be a good idea to wipe it because there’s a good chance that some of the OS files have been corrupted.
  • Restored from backup
  • Performed a Full antivirus scan on the storage server

In our case we received a help desk call a day later from an end user that couldn’t access some of their files. I remoted into their PC and found she was referring to her local My Documents folder. They were all encrypted by the Locky virus. I ran a search of the C: drive to see whereelse the virus had touched. The ODIN file extension was in several directories. My Documents, ProgramData, etc. I noticed the time stamp on the files was one minute prior to the infected files on our storage server. So her workstation is very likely the root of the ransomware infection.

I found this in the registry. I didn’t get a screenshot of the command key value but I looked at it and it was calling Word.exe.

image

I then decided to see how it got into our network. I searched our IPS for events with her computers IP address during the time frame of the infection. And lo and behold I found it. There was a botnet that was allowed from URL logwudorlghdou.info/apache_handler.php

Google searches on the URL logwudorlghdou.info came up with nothing. However, google search on apache_handler.php resulted in the top results being blogs talking about ODIN Locky Ransomware. Here’s an example.

If this was a “High Risk” reputation with URL Category “Bot Nets” then why was it allowed?? We found we had several URL categories blocked but Bot Nets was not on the list. So, we added Bot Nets and another one that looked like a good idea to block named Keyloggers and Monitoring.

I ran a search on the our email archiver and spam firewall. The only message I see sent to our end uers that is questionable is one that has an xls attachment. The from address is dionne.simcoe.195@sunshinecash.com.au so I googled sunshinecash.com.au and found it is some website and has various other websites with the same company name in it (ie. Facebook page) but I was scared to click on it. So then I ran a search for emails sent from that domain. Nothing. So then I ran another serach of emails to anyone but with that subject line. Bam! We received several with the same subject pattern. The emails all contain xls attachments but are from completely different domains. I also looked at the body of the email and all of them have the same body “Sent with Genius Scan for iOS”.

Moral of the story? Education is so important. Train your end users NOT to click on links in emails they don’t recognize! And add as many layers of security that you can. Antivirus, antispam, antimalware, intrusion prevention, firewall, dns protection, and the list goes on.

Migrating Exchange 2007 to 2013–Phase 2

This second phase of my Exchange Migration includes the actual installation of Exchange, configuring our Spam Firewall, Message Archiver, Load balancer and testing. I did this all last fall but just now got around to blogging. Bah! Who has time to blog? But blogs to me are soooooo important to the existence of my IT knowledge so I take this opportunity to pay it forward.

Resources:

  1. Verified we don’t have an existing SPF record by using the Microsoft Sender ID Framework SPF Record Wizard.image
  2. Added my AD account to Enterprise Admins and Schema Admins groups temporarily for Exchange installation. will remove at end of project.
  3. Installed prerequisites on new server
    • Media Foundation service
    • Unified Communications Managed API 4.0
  4. Started Microsoft Exchange 2013 installation wizard to complete remaining prerequisites. Rebooted. Ran Microsoft Updates and installed 9 new updates. Rebooted.
  5. Started installation. It warned that no Exchange 2010 servers were found in the environment so once it preps AD and extends the schema, we will not be able to install a 2010 server.
  6. Setup completed in about 30 minutes.                                                                                                
  7. To access the new Exchange Control Panel browse to https://<servername>/ecp/?exchclientver=15
  8. You can now see that both servers are listed in the servers tab.                                             image
  9. Generated CSR from the new server to upload to Digicert.                                 image
  10. Used it to add new domains to the DigiCert so now the following exist. 
    •    mail.mydomain.com
    •    activesync.mydomain.com
    •    owa.mydomain.com
  11. Uploaded the cert to the Exchange server. Exported it and saved it in a safe location.
  12. Added the Exchange server to Basic > Services with a new VIP that we selected. Used the Microsoft Exchange Server 2013 Deployment document for Barracuda Load Balancer to aid us in creating an http redirect to https because the way we had our old exchange server performing a redirect to HTTPS for owa access is not secure and not supported in Exchange 2013. We will be adding a redirect on the Load balancer cause it has this capability. We ended up having to create two new services. image
  13. Added DNS entry for mail.mydomain.com
  14. renamed default database from “Mailbox Database GUID”  to Admin.
  15. moved Admin database to new location on M: and L: drives by running these commands:
    • Move-DatabasePath –identity admin –EdibFilePath “M:\ExchangeFiles\Databases\Admin” –LogFolderPath “L:\ExchangeFiles\Logs\Admin”
  16. Created new databases to migrate mailboxes to. We decided to keep the existing three databases and just prepend the db names with Exch13_ .
  17. Migrated a test mailbox from 2007 to 2013. Successfully sent mail from the mailbox after migration from internally and externally and received.
  18. Created two new Receive connectors:
    1. Internal Relay – this is for relaying messages from devices (ie. printers) to our internal mail recipients. It has the following security settings:                                                            image
    2. Internet Receive – this is for receiving mail from the Barracuda Spam Firewall which was received from the outside. It has the IP address of the spam firewall on the scoping tab and the following security settings: image
  19. Ran this script to change the activesync virtual directory:
  20. I added activesync.mydomain.com to our internal DNS and pointed it to the internal VIP.
  21. Then added the activesync and mail addresses to our public DNS records:
  22. Also added a new SPF record using the TXT option in our public DNS records. We used this wizard to help us generate a string. 
  23. Message Archiver connection:
    • In Mail Sources > SMTP/IM we added the IP address of the new exchange server in Trusted SMTP servers.
    • On the new server I ran the following PowerShell scripts to create a remote domain:

        Execute the following command to create the remote domain; this command ensures TNEF encoding is disabled:

        New-RemoteDomain -DomainName bma.int -Name “Message Archiver Domain”

        2c. Next, execute the following command to enable auto-forwarding:

        Get-RemoteDomain | Where {$_.DomainName -eq “bma.int”} | Set-RemoteDomain -TNEFEnabled $false -AutoForwardEnabled $true

        2d. Enter the following command to verify the settings:

        Get-RemoteDomain | Where {$_.DomainName -eq “bma.int”} |Format-table Name, DomainName, TNEFEnabled, AutoForwardEnabled

      • In the Exchange Control Panel under Servers > Databases in the properties for each database, on the maintenance tab I added the BMA Journaling account “BMA Journaling”.
      • Created an account in AD to be used as a service account to give the Barracuda access to the mailboxes. 
      • Ran the following powershell scripts to give the new account access:
        • Get-MailboxDatabase | Add-ADPermission -User <service account name> -AccessRights ExtendedRight -ExtendedRights Receive-As, ms-Exch-Store-Admin
        • Get-MailboxDatabase | Add-ADPermission -User <service account name> -AccessRights ExtendedRight -ExtendedRights Receive-As, ms-Exch-Store-Admin
      • Back on the Message Archiver, on the Mail Sources > Exchange Integration tab I clicked Start New Action, selected Folder Sync, created a new server connection with new service account to run Nightly. I did the same for Non-Email Sync.
  24. Ran the ActiveSync test from this website https://testconnectivity.microsoft.com/
  25. Attempted to install the Barracuda Antivirus for Exchange agent version 7.1.7.0. I downloaded the Exchange 2013 agent from Advanced > Exchange Antivirus. However when I install it I get this error. I tried twice and get the same error. I verified the Microsoft Exchange Transport service is running before I start the install. I can see that during the install the installer stops the service. It seems to have trouble restarting it after the install is finished.
    • Day4_1.jpg

  26. Contacted Barracuda support. They haven’t seen this before but found this Microsoft KB2938053 for me that addresses the issue. And here’s another article explaining it. I ran the powershell script and installed the antivirus agent successfully after that. Wahoo! We now see both servers under antivirus:                                                                                                  image
  27. Created new services on the Load balancer under Basic > Services to allow communication to mail.mydomain.com on port 587.image
  28. Attempted to migrate my own mailbox over. It failed with error Active Directory property ‘homeMDB’ is not writeable…image
  29. Apparently this is common on accounts that were at one point Domain Admins. Checked the inherit permissions option in my AD account Security > Advanced. Started the migration again. One problem I see with the EAC is that the migration status just shows that it’s “syncing” and not what percentage it’s on. So I ran the Get-MoveRequestStatistics powershell script.image
  30. Migration completed successfully and since I had my Outlook open at the time, I tried to send a new message and got this pop up (actually it was hiding I had to find it on my task                                                                                  image
  31. Configured the limits on the databases. The defaults were preventing me from migrating any mailboxes that were already over size.
  32. Migrating Managed Folder Policies… apparently managed folder policies don’t exist in Exchange 2013. So I had to use this article to aid me in migrating. Basically I created new Retention Policy Tags, then created a new Retention Policy and applied it to the databases.
    • Old server settings:
    • image
      • Ran the following scripts to create the Retention Policy Tags:

          New-RetentionPolicyTag CompanyName-Calendar -Type Calendar -RetentionEnabled $true -AgeLimitForRetention 1095 -RetentionAction PermanentlyDelete

          New-RetentionPolicyTag CompanyName-DeletedItems -Type DeletedItems -RetentionEnabled $true -AgeLimitForRetention 30 -RetentionAction PermanentlyDelete

          New-RetentionPolicyTag CompanyName-EntireMailbox -Type All -RetentionEnabled $true -AgeLimitForRetention 365 -RetentionAction PermanentlyDelete

      • Created a new Retention Policy and assigned the new tags:
      • image
  33. Modified authentication for OWA under servers > virtual directories. Changed from “Use forms-based authentication” with “Domain\username” to “Use forms-based authentication” with “User name only
  34. performed an iisreset
  35. We’re getting this error when trying to access public folders from Outlook clients if the user has a mailbox on the new server. image
  36. In order to resolve this we made the following changes:
    • On the old 2007 server we changed the authentication from Basic to NTLM.               image
    • On the new 2013 server we changed the authentication from Negotiate to NTLM. We should change it back to Negotiate once we remove the 2007 server from our environment though. Old settings in first screenshot and new ones in second screenshot. image
    • image
    • It was resolved after I rebooted the 2007 server over the weekend.
  37. We also added the new server to the Barracuda Outbound send connector. image
  38. We also added the IP address of the new server to the Trusted IP’s under Basic > Outbound on the Barracuda Spam Firewall
  39. The C: drive on the new Exchange server was already getting pretty full. So to prevent future problems, I shutdown the VM, increased the C: volume from 60 GB to 120 GB, turned back on the VM and extended the disk in Windows Disk management. ​
  40. Upgraded to Exchange 2013 CU9 and rebooted for good measure.
  41. Imported Transport Rules – I was alerted by a user that their scanned in email was going to their junk folder. I remember this happened quite a bit a long time ago and I created a Hub Transport Rule to avoid this. This made me wonder if transport rules were being used on the mailboxes migrated to new server. I tested this by sending an email to my external personal account. I noticed our “external notation” disclosure on the footer of the email was missing. So I knew for sure that transport rules aren’t being used. Rather than recreate them all on the new server, I performed the steps found in Microsoft KB 2846555 to import them.
  42. On the 2007 server, I ran the following Powershell script:

    Export-TransportRuleCollection -FileName c:\ExportedRules.xml

    On the 2013 server, I copied the exported file and ran the following Powershell scripts:

    [Byte[]]$Data = Get-Content -Path “C:\ExportedRules.xml” -Encoding Byte -ReadCount 0
    Import-TransportRuleCollection -FileData $Data

  43. We later found that even though I imported the transport rules, our annotation wasn’t being added to the outbound emails. Our consultant helped me  out and found that the Remote Domain named “Default” had the “IsInternal” property set to “True”. He ran this script and now it shows False. We tested and the annotation is being appended now. Yay!     Set-RemoteDomain –Identity default –IsInternal $false
  44. Final Stuff:
    • Removed the old 2007 server from the send connector named Barracuda Outbound.
    • Our consultant noticed there was an error I the event log that indicated there was an authentication issue (forgot to document the event entry). He added Authenticated Users in the ASDI Edit to the Client Proxy and the errors went away. image
      • I realized I hadn’t migrated the resource mailboxes to the new server. So I tried to move them via the Exchange Admin Center.

        I kept getting this when I tried to select a Resource mailbox in Exchange 2013. I just want to migrate these to the new server. So I tried just running the command shell and it worked. Used this: New-moverequest –identity “ResourceMailboxName” –targetdatabase “Exch13_General” –baditemlimit 10image

      • Under Server Configuration > Mailbox, we right-clicked each of the the OLD databases and removed the old 2007 server name. Then right-clicked each OLD storage group and removed them. There’s only one remaining and that is public folders. Once we are ready to remove all we can remove the storage group for it.
      • We set the Offline Address Book to the new one through the ECP under Servers > Databases. Edited each database, under Client Settings we set the Offline address book to “Default Offline Address Book (Ex2013)”.
      • We then went to Exchange 2007 EMC under Organization Configuration > Mailbox > Offline Address Book, right-clicked the “Default Offline Address book” and deleted it.

       

      Remaining Steps to decommission the old server:

      • Delete all of the public folder data. Then go into ADSI Edit and delete Public Folder SG found under Configuration > Services > Microsoft Exchange > <DomainName> > Administrative Groups > Exchange Administrative Group > Servers > <Exchange2007servername> > InformationStore.
      • Get rid of dependencies on Send Connectors
      • Get rid of dependencies on Receive Connectors
      • Change DNS records (internal and external) for OWA.
      • Uninstall Exchange 2007 using Add/Remove Programs on old server.

DONE!!

Well… pretty much. The remainder of the adventure included getting all the applications utilizing SMTP to point to the new server.

     

Exchange 2013 Log Files

Okay, so I’ve been totally slacking on my posts. Especially regarding Exchange 2013. I haven’t even written my “Migration Part 2” post. I’ll try to get that done soon. In the mean time…

Running a new Exchange 2013 server? Are you finding that your C: drive is filling up and it’s not apparent what is the cause? Me too! For years I’ve been using this handy dandy tool called WinDirStat that helps me find what files are consuming drive space on my servers. I’ve been using WinDirStat on our Exchange 2013 server for three months cause I knew space was creeping up to 80% consumed so I wanted to see what was causing this. At the beginning of my search I found that C:\Program Files\Microsoft\Exchange Server\V15\Logging was pretty large and seemed to be growing. After some light reading I learned that it should plateau and it did at just over 20GB and has been stead for the last two months. But my C: drive keeps filling but WinDirStat isn’t reporting any new large amounts of data. Hmmm… interesting.

I decided to look at WinDirStat and see if there’s some setting for viewing hidden files or some other setting that might reveal the culprit. I noticed there’s an option to view “Unknown” files. So I selected this option. Voila! There it is! 37GB of <Unknown> data. Well, what the heck is that? So I did some searching and found this article on the WinDirStat blog. And after reading it, I got to thinking… have I ever tried running the program as Administrator (you know, right-click > Run as Administrator). No, I don’t think I have. Why not? I don’t know. I’m in the habit of installing Setup files with run as admin but I guess it’s not everyday that I run application exe’s with run as admin. So, I try it… double voila!

Mystery solved. What are the unknown files that are consuming the C: drive? Yup, you guessed it (or maybe you didn’t), C:\inetpub\logs\LogFiles. Geez. I feel like a doofus. I should have guessed this because I have other servers that I run scripts occasionally to cleanup this old data. But I wasn’t sure if there were any precautions I needed to take prior to purging so I found this article titled Exchange 2013 Logging: Clear out the log files. I manually deleted all but the current month’s log files and will create a script to clean up on a regular basis.

Group owners – why can’t I assign them?

So it’s been months since I finished migrating to Exchange 2013. Today I get a call from one of our management staff. She’s trying to add a new member to a group that she used to be able to add members to.

I open the group in ECP (aka EAC) and find that there is no owner assigned. So I click the plus sign and add her. When I click Save I get this nice message.

Group_Error

Well ain’t that grand. I’m an Exchange Org Admin and I can’t add an owner. A quick little google search gets me to this article. Ah thank goodness for PowerShell. So I run this command and it prompts me that “the object must be upgraded to the current Exchange version.” I select Yes and voila! She’s now an owner again.

Set-DistributionGroup “USGName” -BypassSecurityGroupManagerCheck -ManagedBy Owner

Easy peasy. done.

 

Windows 10 Upgrade Experience 

So after ensuring I had a good backup, I decided to dive in and upgrade to Windows 10.

My computer?

  • HP Pavilion
  • Windows 8.1

Schedule

Install

WorkingOnIt
After the install started it went into configuring upgrade for about 5 minutes. Once it reached 100% it rebooted and displayed this screen.

20150801_173417409_iOS

It rebooted a couple times. I went and got a glass of water, got distracted by something and came back just as it was finishing. I was presented with a brand new shiny interface.

WelcomeBack

This picture doesn’t do it justice but I particularly like the big Window. I think it’s a nice touch.

Welcome

And upon logging in I can see I have the “Start” menu back. Yay! Now I can get rid of the 3rd party app I had installed a couple years ago.

I haven’t had much time to explore the new features but next I’ll definitely be reading this article on how to secure my privacy. I hear the upgrade count is up to 100 million and climbing!

Windows 10, a cozy place to go

When I first heard that windows 10 was going to be a free upgrade for those who currently own Windows 7 or Windows 8 or 8.1 I wondered what the catch was. Why would Microsoft be giving away yet another upgrade for free? This morning on my way to work there was a highlight on the radio where the journalist talked about windows 10, the highlights that most people would appreciate and why Microsoft will be giving away for free. The Windows operating system has historically been Microsoft’s moneymaker. But as the years pass it seems that consumers have less faith in Microsoft. PCs are continually plagued with viruses and malware exploits. And consumers would often wonder why should I pay hundreds of dollars for this operating system when the one that I currently own works just fine? But those consumers don’t realize that having an older operating system leaves their computer to be more vulnerable. So if Microsoft lets their customers upgrade for free, they will allow them to see the benefits of the new features of Microsoft’s operating system and their computer will be more secure leaving happy customers. Can Microsoft redeem itself?

There have been several articles on the new features of windows 10 so I won’t get into the detail. But, with every new operating system there are security enhancements. The consumer highlights will be the come back of the start menu, Cortana (Microsoft’s answer to Siri) and a new browser called Edge. Bringing back the start menu alone I think will make consumers believe that windows 10 is a cozy place to go, a familiar face, a reason to upgrade.

Will I be upgrading? I think so. On both my home computers? I’m not sure about that. I will upgrade one and see how it goes. Will you be upgrading?

Migrating Exchange 2007 to 2013 – Phase 1

We’ve been putting off migrating our Exchange 2007 server for quite some time. It’s been running great. I rarely have had to troubleshoot any problems (maybe twice a year). I took the training provided by Microsoft from one of our local training facilities. The class was titled Core Solutions of Microsoft Exchange Server 2013. But you know that was two years ago! So my new found knowledge went unused and most of it has been filed away in the archives of my brain. Luckily the company decided they would pay for a consultant to come in and help. That brings a lot of relief to me. Yes I know Exchange, but I’m not an expert. Sometimes I consider myself a Jane of All Trades and this is one of those technologies that fits under that umbrella. But regardless, I’m excited!

Phase 1 – Prepare the environment

I’m using a Dell M610 blade server as the new Hyper-v Host for housing our new Exchange server. Yes, I use Hyper-V. When we jumped on the virtual bandwagon years ago we decided to go with Hyper-V because of its cost effectiveness. You may want to skip to Phase 2 if you’re not interested in how I configured the Hyper-V host.

Server has the following:

  1. 2 x Intel Xeon x5672 @ 3.2 GHz
  2. QLogic Fibre Channel mezzanine card.
  3. 32 GB Memory (4 x 4GB DIMMs)
  4. Installed Windows Server 2012 R2 Standard.
  5. Configured a static IP address on one of the NICs, joined the domain and activated Windows.
  6. In Windows Firewall I made the following changes:
    • Allowed File and Printer sharing for the Domain profile only. This allows me to ping and gain remote access to the server.
    • We use a Dell Compllent for SAN storage so I created new Inbound rule on Windows Firewall called “Compellent Server Agent” to allow communication through ports 27355 and 8080. This allows the agent to run without turning off the firewall.
  7. Ran Windows Updates.
  8. Downloaded and ran the latest Dell Server Update Utility v15.04.00 to update drivers and firmware.
  9. Installed BgInfo, enabled Remote Desktop, and configured SNMP.
  10. Configured zones and alias on our Brocade SAN switches.
  11. In the Compellent System Manager, I created the server and mapped a new 500 GB volume.
  12. Installed MPIO​ and .NET 3.5 (which includes .NET 2.0) since it’s a requirement for the Compellent agent.
  13. Opened MPIO from Control Panel, added the Compellent controller to the Discovery tab and rebooted.
  14. Installed the Compellent Server agent and added the server to the Enterprise Manager.
  15. Rescanned disks in Disk Management, initialized the volume and added a new Simple Volume, labeled it G:
  16. Installed the Hyper-V role with the following settings during setup and then rebooted.
  17. I configured the virtual switch on the 2nd Ethernet adapter, Ethernet 2 (no ip address assignment necessary)image
  18. I set the default stores to the G: drive. image
  19. Installed our Barracuda Backup Agent and SCEP (system center endpoint protection) client.
  20. Added the new server to Virtual Machine manager and it required yet another reboot. Geez! image
  21. Rebooted AGAIN!! VMM stated that”Warning (26211) A restart is required to complete claiming of multi-path I/O devices on host hostname.domain.com. Recommended Action
    Restart the machine.” Ha! Reboot.
  22. image
  23. Rebooted. Ran Windows updates again since it has a new role. Installed more updates and rebooted.
  24. Ok now I’m finally to the point where I can create the VM. So I used Microsoft Virtual Machine Manager since I already have VM templates configured.
  25. I configured the VM to have the below settings.
  26. Configured a static IP address, joined the domain and activated Windows.
  27. Ran Windows Updates.
  28. Copied the Exchange 2013 SP1 .ISO file to the new VM

VM Settings:

  • 24 GB Memory
  • 4 x virtual processors
  • Drives:
    • C: – 60 GB – operating system
    • M: – 250 GB – databases
    • L: – 100 GB – logs
  • no high availability since this Hyper-V host is not in a cluster.

Done! Ready for next week!