1.
Upgrading
from Rovius
CloudPlatform to
Apache CloudStack
Andrija Panic
CloudStack Collaboration Conference 2022 / #CSCollab2022 / 14-16 Nov 2022 / Sofia, Bulgaria

2.
About me,
myself and I
Cloud Architect @ ShapeBlue
virtualization for last 15+ years
Involved with CloudStack since
version 4.0.0-incubating
Apache CloudStack project committer
and PMC member
CloudStack Collaboration Conference 2022 / #CSCollab2022 / 14-16 Nov 2022 / Sofia, Bulgaria

3.
Acronyms used
ACP =
Accelerite CloudPlatform
(Rovius CloudPlatform)
ACS =
Apache CloudStack
CloudStack Collaboration Conference 2022 / #CSCollab2022 / 14-16 Nov 2022 / Sofia, Bulgaria

4.
What am I
going to see
today?
High level overview of the whole
migration process
Some detailed steps
Some gotchas and what to look for
A lot of WHAT and HOW
A walk through a complete upgrade
project ( sample customer)
CloudStack Collaboration Conference 2022 / #CSCollab2022 / 14-16 Nov 2022 / Sofia, Bulgaria

5.
Because such thing does not exist
Why not
step-by-step
guide?
CloudStack Collaboration Conference 2022 / #CSCollab2022 / 14-16 Nov 2022 / Sofia, Bulgaria

6.
Now seriously
Each customer environment is different
Different ACP version
Different target ACS version
Different DB data, different DB schema,
patches, etc.
Different auxiliary components
Cloud Portal Business Manager
LDAP/NATIVE authentication
etc.
Requires judgement calls most of the
CloudStack Collaboration Conference 2022 / #CSCollab2022 / 14-16 Nov 2022 / Sofia, Bulgaria

7.
Challenges
Ensure successful DB upgrade
Ensure minimal DB deviation from a
clean ACS install DB
Proper testing (functional, UAT)
Ensure proper night sleep after the
upgrade
CloudStack Collaboration Conference 2022 / #CSCollab2022 / 14-16 Nov 2022 / Sofia, Bulgaria

8.
Required tools for a successful upgrade
A virtual machine with target ACS mgmt. installed (for sandboxed DB upgrade)
dbForge software:
Data compare & Schema compare tools (www.devart.com)
For functional/UAT testing
Optional: ACP binaries
to spin a new test env, upgrade to target ACS version, run Marvin/smoke tests.
CloudStack Collaboration Conference 2022 / #CSCollab2022 / 14-16 Nov 2022 / Sofia, Bulgaria

9.
7-layer OSI networking model ?

10.
25% insanity - - Oscar Levant
75% sanity
Understanding of ACS upgrade process
Understanding of ACS DB structure and data
MySQL and SQL knowledge
Good judgement skills are critical
Other stimulants
Headache painkillers, coffee, music && in late hours
A bit of luck
CloudStack Collaboration Conference 2022 / #CSCollab2022 / 14-16 Nov 2022 / Sofia, Bulgaria

11.
Are you Ready ?

12.
A brief history of ACP and ACS
Launched in May 2010 as Vmops, rebranded to Cloud.com.
Acquired by Citrix in 2011, donated to the Apache Software Foundation in 2012
Citrix maintained their own commercial fork -
In January 2016 Citrix sold CloudPlatform to Accelerite
Development of Apache CloudStack continue independently in an open-source community.
Version at time of writing:
Apache CloudStack LTS 4.17 released June 2022 (4.16 LTS November 2021)
Accelerite CloudPlatform latest release 4.11, based on Apache Cloudstack version 4.10.
CloudStack Collaboration Conference 2022 / #CSCollab2022 / 14-16 Nov 2022 / Sofia, Bulgaria

13.
Differences between ACP and ACS
While essentially the same product there are some significant differences
ACP 4.11 is based on ACS 4.10
ACP 4.7 based on ACS 4.6 with some tables indexes from ACS 4.8, 4.9 and even 4.10
guest_os guest_os_hypervisor
Feature set mismatch
ACP developed some additional features (for their customers) that are not in the upstream (ACS)
Different implementations of same things in ACP and ACS
CloudStack Collaboration Conference 2022 / #CSCollab2022 / 14-16 Nov 2022 / Sofia, Bulgaria

14.
dbForge schema compare
https://www.devart.com/dbForge/mysql/compare-bundle/
SCHEMA compare
Compare table structures (field lengths, indexes, keys, columns names, etc.)
Take the clean installation ACS schema as the source
Take the ACP schema as the destination (ACP upgraded to ACS)
updating the destination DB to be like the source DB
Thus, important to not swap ACP and ACS DBs
Make manual changes as needed
Export the script
Why SCHEMA changes are important?
ACS CODE expects certain DB schema - as simple as that
CloudStack Collaboration Conference 2022 / #CSCollab2022 / 14-16 Nov 2022 / Sofia, Bulgaria

15.
dbForge schema compare demo

16.
dbForge data compare
https://www.devart.com/dbForge/mysql/compare-bundle/
DATA compare
configuration, roles, role_permission, hypervisor_capabilities, guest_os_hypervisor, guest_os*
Take the clean installation ACS as the source
Take the ACP as the destination (ACP upgraded to ACS)
updating the destination DB to be like the source DB
Thus, important to not swap ACP and ACS DBs
Make manual changes as needed
Export the script
Why DATA changes are important?
ACS CODE expects certain DATA in certain tables - as simple as that
CloudStack Collaboration Conference 2022 / #CSCollab2022 / 14-16 Nov 2022 / Sofia, Bulgaria

17.
dbForge data compare demo

18.
Risks with DB changes
Shortening column length
Is there data that would be truncated
Lowering values inside some columns
hypervisor_capabilities
dbForge tools
hypervisor_capabilities
wrong hypervisor_type hypervisor_version
False differences due to comparing unneeded columns
CloudStack Collaboration Conference 2022 / #CSCollab2022 / 14-16 Nov 2022 / Sofia, Bulgaria

19.
HIGH LEVEL UPGRADE STEPS

20.
High level upgrade steps
(sandboxed) copy-of-production-DB upgrade
Usually requires pre-upgrade database troubleshooting and cleanup
This is done in sandboxed environment (e.g. inside a VM, network-isolated from HV hosts/storage)
Ensure easy rollback in case of failed DB upgrade (e.g. VM snapshots)
Clean up databases schema and data after the upgrade
Required for various reasons
Time consuming, prone to mistakes, rinse and repeat
Test
Functional testing possible only in the test env
CloudStack Collaboration Conference 2022 / #CSCollab2022 / 14-16 Nov 2022 / Sofia, Bulgaria

21.
Production DB upgrade
Management server build (target OS + target ACS version)
Install/configure target mysql version on the same server
Create mysql user* for cloud/cloud_usage database
db.properties
db.cloud.encrypt.secret db.properties
db.cloud.password db.properties
-setup-
vm_template
Perform pre-upgrade database cleanup (once you have the SQL scripts)
Start cloudstack management server, tail f the logs, confirm DB upgrade
CloudStack Collaboration Conference 2022 / #CSCollab2022 / 14-16 Nov 2022 / Sofia, Bulgaria

22.
Production DB upgrade
Ensure doing frequent VM snapshots
)
db.cloud.encrypt.secret db.cloud.password
java -classpath /usr/share/cloudstack-common/lib/jasypt-1.9.2.jar
org.jasypt.intf.cli.JasyptPBEStringDecryptionCLI decrypt.sh input=<ENCRYPTED-VALUE>
password=<management-key-value> verbose=true
Management server setup
cloudstack-setup-databases cloud:<cloud db password>@<cloud db host> -i <management server IP
address> -m <mgmt-key> -k <database-encryption-key>
Simulate system VM templates are in place
TLDR/W
CloudStack Collaboration Conference 2022 / #CSCollab2022 / 14-16 Nov 2022 / Sofia, Bulgaria

23.
PRE-UPGRADE database cleanup
Depend on the ACP version, custom patches, and additions of guest OS records
Usually boils down to:
guest_os
Adding/removing keys/indexes/other schema changes
ACP 4.7 has a lot of keys/indexes improvements from ACS 4.8, 4.9 and even 4.10,
but is based on ACS 4.6 version
so that the ACS SQL upgrade scripts can run successfully (from 4.6 and onwards)
CloudStack Collaboration Conference 2022 / #CSCollab2022 / 14-16 Nov 2022 / Sofia, Bulgaria

24.
PRE-UPGRADE database cleanup
guest_os
Records/values started drifting heavily in last few ACP/ACS versions
We need to remove records with IDs that ACS upgrade is trying to inject otherwise,
the upgrade of DB will fail (record with the same ID already exists)
this requires updating references in various other tables
We need to work out on which ACS version is the current ACP version based on
Done by understanding different failures of the DB upgrade
Done by checking the ACS SQL upgrade files
e.g. ACP 4.7 is based on ACS 4.6, ACP 4.11 is based on ACS 4.10
CloudStack Collaboration Conference 2022 / #CSCollab2022 / 14-16 Nov 2022 / Sofia, Bulgaria

25.
POST-UPGRADE database cleanup
Both schema and data cleanup needed
to ensure that ACS will not throw exceptions and various functionalities will not be broken
older ACP data/schema is there
Role permissions records are completely missing
Hypervisor capabilities differ
PrivateGw network_offering
unique_index ssh_keypairs
duplicate records need to manually remove older one of each duplicate key
etc.
CloudStack Collaboration Conference 2022 / #CSCollab2022 / 14-16 Nov 2022 / Sofia, Bulgaria

26.
Guest_OS

27.
Sample customer project

28.
Thanks!
Q&A ?