Operating system security (a brief)

IBM Global Business Services
OS Security
OS Security March-2007 © 2007 IBM Corporation

Objectives
 What is OS Security ?
 OS security breakdown
 Security in different OS environments
© 2007 2 OS Security March-2007 IBM Corporation

OS security is important
 „ Fundamental basis of most systems
 „ Control hardware/software resources
Introduction

Road Map
 OS security basics
 Security For User Accounts
 File Systems
 Networking
 Architecture
 Authentication
 Unix Authentication
 PAM
 Windows Authentication
 GINA
 Access Control
 Impersonation
 Logging And Auditing
 API
 Memory Protection
 Buffer Overflow
 SAP On Windows
 SAP User Security
 Best Practices On
SAP-Windows
Environments
 Best Practices On
SAP-Unix/Linux
Environments

OS security basics
Security is typically achieved based on
 „ separation and controlled sharing
 Separation applies to (everything)
 „ Internal resources, typically process memory and
 OS data structures
 „ User resources, typically files
 „ System resources from normal users
 Sharing with access control protection
Contd.

OS security basics
 Separation and controlled sharing require
 „ Memory protection
 „ Subjects (users and processes) identification and authentication
 „ Objects (files and other resources) identification
 „ Access control for all

Accounts
 User identification and authentication
 „ Based on account identifier and credentials
 Accounts hold user rights and privileges
 „ For access control
 Accounts may belong to groups
 „ Group has associated rights and privileges
 „ Group-based access control

UNIX accounts
 Each user has an account
 „ On a computer or an NIS(+) domain
 „ Non-human users are for system processes
 Account has name and password
 „ Authentication based on hashed password
 „ OS supports password strength, aging policies
 „ Add-on supports for other mechanisms such as Kerberos, s/key, etc.
available
 A user may belong to many groups
 „ Has the groups’ rights
 „ But effectively only 1 group at a time

Windows accounts
 Each user has an account
 „ On a computer and/or an Active Directory domain
 „ Non-human accounts are for system processes
 Account typically has name and password
 „ Authentication based on Kerberos or hashed password (for NT compatibility
only)
 „ OS supports password strength, aging policies
 „ Certificates and smartcards are also supported (in 2000/XP, but not
commonly used yet)
 A user may belong to many groups
 „ Has the union of the groups’ rights at any time

Networking
 Most systems allow users network access
 OS tools and services enable these access
 „ Their own security issues
 Required integrated network access are explained later
 „ Integrated domain authentication
 „ Network file shares

UNIX networking
 Traditionally set of r- commands
 „ rlogin, rsh, rcp, etc. and corresponding servers
 „ Host address based authentication
 „ Implicit trust on ports lower than 1024
 „ Send passwords in clear-text if required
 „ Very insecure, should not be used anymore
 The ubiquitous telnet, ftp
 „ Clear-text passwords in basic setup
 More secure tools available
 „ SSH, Kerberized telnet, ftp
 Integrated NFS, NIS(+) explained later

Windows networking
 Essentially similar tools
 „ telnet, ftp with clear-text passwords
 „ SSH, and augmented versions of telnet, ftp more
secure
 Integrated networking explained later
 „ Server Message Block (SMB) based
 integrated domain authentication, file shares access

File systems
 File systems security governs
 „ Access control to files based on subjects
 „ Security of files sharing
 „ Files encryption (if any)
 Files include
 „ Data, program and
 „ Other file-based resources, e.g. system caches, named
pipes

UNIX file systems
 Basically one system with native UNIX format
 Access controls using permission bits
 „ read, write, execute permissions
 „ owner, group or others
 „ E.g. –rwxr-x---
 „ Coarse-grained
 Files sharing using Network File System (NFS)
 „ Machine access to shares is based on IP address
 „ User access to shares based on permission bits
 „ Add-on support for Kerberos auth. available
 No support for files encryption

Windows file systems
 FAT (for backward compatibility)
 „ FAT supports no access control
 NTFS (NT File System)
 „ Access control based on user IDs and file permissions
 „ Basic permissions are Read, Write, Execute, Delete, Change
Permissions, Take Ownership
 „ Standard permissions are basic ones combined
 „ Different permissions to a file can be granted to individual
users/groups using ACL
 „ More fine-grained, flexible than UNIX
Contd.

Windows file systems
 Files sharing using Common Internet File System (CIFS)
 „ Shares are managed in directory (in common with domain management
– more later)
 „ Machine access to shares is based on computer account in domain and
inter-domain trust
 „ User access to shares is based on share passwords or standard ACLs
 „ NT systems use hashed password SMB auth.
 „ Windows 2000/XP use Kerberos authentication
 Encrypting File System (EFS)
 „ Files encryption using random secret keys, which are in turn encrypted
with EFS public keys

UNIX security: Architecture
 Basic UNIX based on monolithic kernel
 Fundamental OS security based on
 „ User id and password
 „ Group id
 „ Process id
 „ File permission bits
 „ Process memory protection

Windows security: Architecture
 Windows (NT/2000/XP) have layered components on top of
a kernel
 Security Reference Monitor (SRM)
 „ Part of the kernel
 „ Handles core of access control checks
 Protected security services include
 „ Win logon process
 „ Local Security Authority (LSA) and policy database
 „ Security Account Manager (SAM) and database
 „ These services perform user authentication, and non-core part of
access control
Contd.

Windows security: Architecture
 Security identifiers (SID)
 „ Represent uniquely each user or group
 Access control entry (ACE)
 „ Contains permissions to an object explicitly denied or granted to a
subject (SID)
 Access control list (ACL)
 „ List of ACE’s for an object
 Security descriptor of an object
 „ Contains is owner SID, primary group SID, its ACL, the applicable
system ACL
 Access token for a logged on user
 „ Contains the user’s SID, primary group SID, etc.

UNIX security: Authentication
 Username and clear-text password
 „ For single computer or NIS(+) domain
 „ System stores (modified DES) hashed passwords
 „ /etc/passwd readable by everyone, or
 „ /etc/shadow readable only by root, or
 „ NIS(+) database
 „ Passwords are hashed before matching
 „ Logged on users are identified by numeric IDs
 „ Passwords are open to dictionary attacks
 Integration of Kerberos and others methods
 „ Pluggable Auth. Module (PAM) for Solaris, Linux
 „ Security Integration Architecture (SIA) for HP/UX

Pluggable Authentication Module (PAM)
Login Telnet Ftp
PAM API
PAM Framework
PAM
Configuration
PAM SPI
UNIX Kerberos Smart Cards

Windows security: Authentication
 NT uses NTLM authentication
 „ NT (MD4) and LM (DES-based) hashed password
 „ Domains integration relies on sending hashed passwords through
insecure SMB protocols
 „ Inter-domain trusts are one-way, non-transitive
 Windows 2000/XP in domains use Kerberos
 „ NTLM supported for backward compatibility
 „ Domains are managed by Active Directory
 „ Integrated Kerberos auth. as domain controllers are KDCs
 „ Enable hierarchical organization and delegation
 „ Inter-domain trusts are two-way, transitive thereby simplifying trust
management
 Logged on users run processes with their access tokens,
basis for access control, impersonation

Graphical Identification And Authentication
(GINA)
Win Logon
GINA
LSA
Shell
Registry
Win Logon Shell
My GINA Registry
GINA LSA
LSA

UNIX security: Access control
 Only discretionary access control (DAC)
 „ Based on file permissions and UID, GID, PID
 „ File has permission bits, UID (owner), GID
 „ File permission bits are r, w, e, and s (later)
 „ A process has real and effective UID and GID
 „ Kernel matches these IDs to control a process’s access to a file
 „ Super-user (root) has all access to everything
 „ Some variants such as Solaris 2.5 or newer have
 ACL systems for more fine-grained controls
 Some experimental systems (e.g. SE Linux) have
Mandatory Access Control (MAC)

Windows security: Access control
 Discretionary access control
 „ Based on subject SIDs and object ACLs
 „ Each object has an ACL
 „ Null ACL or empty means no restrictions or no access
 „ Each process has an access token with its owner SID, group SIDs
 „ Access control checks are matching of access tokens against ACLs
 „ Administrators group can access everything
 „ SRM performs core matching
 Less so discretionary access control
 „ Some system-wide policies applying to subjects, regardless of individual
object’s ACL

UNIX security: Logging and auditing
 Flexible and comprehensive “syslog”
 „ Logging daemon can store locally or on remote server
 „ System processes store relevant information through logging APIs
 „ System administrators can configure what to log, and where to
store logs
 „ However, auditing tools are not natively available in the basic OS

Windows security: Logging & auditing
 The LSA and SRM create logs through the system event
logger
 The LSA logs mostly logon events based on its audit policy
 The SRM logs access check events based on the system
access control list (SACL)
 „ Each object has an SACL
 Logs are stored locally

UNIX security: Impersonation
 Static privileges are often too restricted
 Impersonation allows dynamic changes in a user or process’s
security privileges
 Programs run with its owner or group ID instead of user who runs
them if
 „ Set-UID (suid) bit set, or
 „ Set-GID (sgid) bit set
 Flaws in these programs can be extremely dangerous
 User can impersonate other users by
 „ Running “su” to have an impersonated shell
 „ Running “sudo” to impersonate for a command

Windows security: Impersonation
 No equivalence of UNIX suid, sgid or “su”, “sudo” programs
 But processes frequently programmatically impersonate others
 „ A thread takes on access token of another subject
 „ This access token may be exact copy or variant of a primary access token
 „ Thread gets security privileges of the impersonated subject
 Impersonation is application-controlled, as opposed to administrator-controlled
in UNIX

OS security: buffer overflow
 Example code:
int auth_user() {
char name[32];
printf(“Enter username: “);
gets(name);
/* do authentication */ }
 User enters more than 32 characters
 Variable name gets the first 32 characters
 The rest goes on the program stack
 May override program pointer
 Program then jumps to unexpected code

OS security: memory protection
 Standard process memory protection
 „ Process memory is accessed through page table
 „ No process can normally access another’s memory
 „ Historically for safety, but critical for security
 Buffer overflow
 „ Arguments and program pointer on the stack
 „ Writing beyond the buffer for an argument may overwrite the
program pointer
 „ Careful selection of argument data may get program to execute
malicious code
 „ Compilers and/or operating system can help prevent this from
happening

UNIX security: APIs
 Basic OS supports few security APIs
 „ Essentially user, password, and process management
APIs
 Modern variants support more
 „ E.g. PAM APIs
 Add-on services are relatively common
 „ Kerberos APIs, GSSAPI, OpenSSL

Windows security: APIs
 Windows support
 „ Essential user, password, process management APIs
 „ Graphical Identification and Authentication (GINA) APIs, fairly
similar to PAM, SIA
 „ Security Services Providers Interface (SSPI) similar to GSSAPI
 „ CryptoAPI supports encryption, smartcards

SAP And Windows Security

Protecting the Operating System Users Used in an SAP System
User type User Function and Rights Security Measures
Windows users Administrator The local superuser who has
unlimited access to all local
resources.
Change the user name and hide its
password. Create other users for
administrative tasks and limit their
rights to those tasks for which they are
used
Guest A local guest account who has guest
access to all local resources.
User type User Function and Rights Security Measures
SAP system users <sapsid>adm The SAP system administrator who has
unlimited access to all local resources
related to SAP systems.
• Change its password regularly.
• Restrict its access rights to instance-specific
resources for the SAP system only.
SAPService<S
APSID>
A special user who runs the Windows
services related to SAP systems.
• Cancel the user’s right to Log on locally.
• Restrict its access rights to instance-specific
and database-specific resources only.

An Windows Environment For SAP Security Should Encompass
Security Of
1. Data Relevant to the SAP System
2. Database Files
3. Protection for Dynamically-Created Files
4. Protecting Shared Memory
5. Defining Start and Stop Permissions
6. Secure Using Windows Trusted Domains

An UNIX/Linux Environment For SAP Security Should Encompass
Security Of
 Protecting Specific Properties, Files and Services
 SUID/SGID programs
 Password file (passwd)
 BSD services rlogin and remsh/rsh,
 Services such as Network Information System (NIS) or Network File
System (NFS)
 Protected SAP System Directory Structures Under UNIX/LINUX

Operating system security (a brief)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Operating system security (a brief)

Similar to Operating system security (a brief) (20)

Recently uploaded

Recently uploaded (20)

Operating system security (a brief)

Editor's Notes