BIP Platform Description

Detailed technical description

General description

The Belarusian Information Platform (hereinafter BIP) is a set of software and hardware solutions designed to build scalable distributed cloud information systems of various levels of complexity.

Components of a cloud information system: building an IS on the BIP platform requires a certain set of computer hardware, system software, and organization of communication channels between its components. Depending on the purpose and size of the system, you can choose hardware of the required capacity: start with practically a single computer and scale up the number and characteristics of equipment as the need for computing power grows.

The cloud-based approach allows the platform to be moved to the data center of a cloud storage provider, yielding significant savings on building your own data center and the staff to maintain it.

1. Information resource instead of "fixed requirements"

Unlike custom software that implements a specific set of requirements and, after commissioning, performs only a fixed set of capabilities built into the project, IS built on the platform can be expanded with new capabilities as they grow.

For this purpose, the concept of an "information resource" (IR) has been developed — a software and hardware module operating within the platform's technologies. All service operations (security, operation log, blockchain mechanisms) are performed by the platform's service components; the specialized IR software performs the business logic required by the customer.

2. Components of a cloud information system

A distributed cloud IS consists of the following main components. Each is considered separately, but only together do they form an integral system for processing large data arrays by many users with online access to up-to-date data.

Fig. 1. BIP cloud architecture: workstations, application servers, database server, and file server

• Database Server (DB) — the key component, designed for data storage.
• File Server (FS) — storage of files attached to IS objects.
• Application Server (AS) — there are as many of them as required to meet users' computing power needs. Currently more than 10 AS of varying capacity are in operation.
• User Workstation — any workstation that can run one of the standard browsers and connect to the IS over a secure communication channel from the office or from home.

Any cloud system consists of the above components and/or their analogues.

Platform system stack: Linux OS + PostgreSQL DBMS + Apache/nginx HTTP servers + the platform's BCLIP execution system. The specific Linux distribution is selected to meet the customer's requirements — the platform is not tied to any single distribution.

2.1. Database Server (DB)

Purpose. The DB is the "heart" of the IS, since data storage and processing is its key function. Storage reliability and operational speed of data access are the main indicators of the system's success.

Hardware requirements. Separating the DB into a distinct component allows the use of equipment of varying capacity. Practically any modern computer can serve as an entry-level DB. More powerful systems include dedicated storage subsystems, clustering and replication, and distribution of processing across several computers in geographically distributed data centers — data safety is guaranteed even up to the destruction of the data center.

System software:

• Linux OS — the base OS for database and application servers, also recommended for workstations. An open alternative to proprietary operating systems with no vendor lock-in. Any OS capable of running the DBMS software is acceptable.
• PostgreSQL DBMS — one of the most common and powerful database servers, widely used by the global scientific community for processing a very wide range of data types of extremely large volume. Used by our company since approximately 2000, chosen in particular to ensure licensing cleanliness of solutions.

Universal integrated database schema was developed by the company specifically for building distributed IS for collaborative work by users of various companies based on common master data with global objects and the ability to allocate information blocks only for employees of a particular enterprise. This made it possible to:

• use unified global data — for example, the ICD-10 reference book for physicians and the HS (foreign economic activity commodity nomenclature) for goods accounting;
• for each enterprise, form data structures accessible only to its employees (fixed assets, vehicles, etc.);
• allow developers to use a unified set of APIs to create various application projects under unified rules with a unified technology for controlling data changes; this increases development speed and reliability, consolidates the work of developers, and makes it easy to extend the set of IRs;
• pass a single security certification of the IS for all projects included in it.

Guaranteed data retention technologies

• Database backup — the standard approach in case of disk space destruction. Typical mistakes: copying to another folder on the same disk (when the disk is destroyed, the copy is lost too; a RAID array is not indestructible) and replication at a frequency inadequate to the update rate. The copy should be placed on removable media in a remote location or in a geographically remote data center. A backup notification system and a person responsible for control are desirable. The classical technology requires at least three copies — for terabyte-scale databases, this requires complex technologies and expensive devices.
• Online database replication — a copy is maintained practically online; the backup database is switched to the primary DB mode within a few minutes, ensuring nearly 24/7 operation. Both standard PostgreSQL replication and a technology developed by our company for parallel writing of data to two DBs are available. Online replication does not replace backup, but ensures 24/7 operation.
• Separation of current and archival data — with a large volume of accumulated data, current and archival data are separated: the archive is stored on slower media and even removable devices to reduce operating costs.

All variants of guaranteed retention are provided in every IS developed by the company. Users of cloud IS are given a 15-minute data retention guarantee: in case of media destruction, data is recovered with a loss of at most the last 15 minutes before the "disaster".

2.2. Application Server (AS)

Purpose. The AS is the "brain" of the IS, processing user requests. Together with the DB, it ensures responsiveness: data entry, modification, and report generation.

Hardware. Any modern computer as an entry-level AS. Powerful systems — clustering and load distribution among several computers in distributed data centers, which guarantees operability under DDoS attacks on one of the data centers.

Software: Linux OS; Apache or nginx HTTP servers; the BCLIP execution system for running CGI modules.

2.3. File Server (FS)

In addition to data in tables, information is stored as files "linked" to IS objects: contract texts, waybills, graphic images. Traditionally these are BLOB fields, which have drawbacks:

• increase processing time when saving (more disk and RAM operations compared to direct file processing);
• reduce processing "transparency" — the DBMS behavior depends on the version and the planner, with no ability to modify the system's behavior.

The platform provides a separate IR "file server": table T185 with file metadata + dedicated disk space for the files themselves. Rejecting BLOB fields provides:

• full transparency of the file processing process and programmatic resources for control and management;
• flexibility in storage placement — from a directory on the AS to a separate software and hardware complex with its own hardware requirements and data retention procedures.

2.4. User Workstation

The development environment allows creating applications with three interface variants:

• Text — without a graphical shell, minimal workstation requirements, highest performance and fault tolerance. Work via terminal sessions; when hosted on the AS — SSH access: minimal communication channel requirements, maximum reliability of remote work.
• Graphical — the GTK library, full GUI. Drawback — orientation only to OSs supporting this library (several Linux versions).
• CGI/WWW — HTTP applications. Combines the reliability of the text interface (no unreliable graphical components) with the capabilities of dHTML, CSS, and JSON. The only drawback — access only via a browser, which is simultaneously the main advantage:

• the application is automatically cross-platform — the only workstation requirement is an installed browser;
• the transport layer is provided by industrial HTTP/HTTPS and TCP/IP protocols;
• all UI variants from the internet world are available, supported by industrial (non-proprietary!) HTTP and CSS protocols;
• the base browser is Firefox, with testing also for Opera and Chrome. Strictly documented protocol capabilities guarantee operability on all platforms.

Workstation software: Linux OS (an open alternative to proprietary operating systems). For text applications — minimal hardware requirements.

3. Communication channels

For applications with a unified database, communication channels to the DB and/or AS are required. Two main data flows: HTTP between the browser and the AS, SQL queries to the DB over TCP/IP. TCP/IP is also used when exchanging with equipment (for example, processing data from analyzers in the "Laboratory Information System"). Traffic types and volumes are taken into account.

Three levels of organization:

• Enterprise local network — all resources (DB, AS, workstations) are on the local network, with traffic inside it. From the simplest hub to complex fiber-optic networks with routers. Connecting remote workstations is economically impractical — the cost and complexity of support grow non-linearly.
• Corporate intranet — enterprise local networks are combined by global links. External channels appear: reliability issues, bandwidth calculation, alternative channels for 24/7, channels for mobile workstations. Support complexity grows almost geometrically.
• Internet / provider data center — the entire range of IS growth issues is solved by the operator: unified workstations to a single standard, 24/7 support, workstation access via a 3G modem as the primary or backup line, connection of all enterprises to a single scheme without building an intranet, emergency access in case of local network/router failure.

Interaction protocols

Fig. 2. Network protocol levels and a separate set of file formats

Application layer (high-level protocols): HTTP / HTTPS, WebSocket, FTP, streaming database and file replication.

Transport layer: TCP/IP, UDP.

Network / data-link layer: Ethernet, WiFi.

Low-level interfaces and protocols: COM (RS232, RS485), PRINTER, USB.

File formats (separate from protocols): TXT, JSON, XML, DBF, HTML. The use of these formats when exchanging data with third-party systems is described in section 12 "Organizing interaction with third-party IS".

4. Cloud IS as a set of information resources

The BIP platform is a set of IRs, each solving a specific information processing task in a common data space with many users. The concept of "IR" was introduced to replace the concept of "IS": an IS solves a specific range of tasks for a specific group of users, while BIP allows creating a set of information services with refinement of existing ones and creation of new ones as needs arise. The technologies allow developing the IS using the Agile methodology; services can be created, maintained, and developed by different development teams for different classes of users.

Methods of service development:

• creating new versions of business process software modules with access provided to the customer-user;
• customizing screen and report forms to user requests without changing the software module (report header per the customer's wishes, etc.) — reduces requirements for the development team and increases reliability due to fewer modules;
• maintaining local master data independently by the customer: global reference books (ICD-10, HS) are filled once and available to all; local ones (warehouse accounting cards) are filled by the customer under their own rules, without a new service version;
• organizing interaction with existing IS via REST API;
• placing the IS in a cloud data center and working over HTTP with a dHTML interface allows providing access not only to corporate divisions, but also to third-party users under the SaaS model — flexible pricing.

Storing source code and settings in the protected database. The protected database stores not only information but also the very logic of the system: the source code of the business logic, as well as separate configuration settings — individually for different enterprises within a single cloud and individually for workstations within a single enterprise, document templates with different text variants for different enterprises, and workstation interface settings. The source code is compiled directly from the database. Since a standard browser is used on workstations, the interface and business logic are updated at the client "on the fly" — without any additional actions to deploy updates. This provides centralized version management and access control.

Built-in documentation system — a system integrated into the platform for storing all documentation in a unified database with information provided according to access rights. Data is organized in a tree: sections are numbered, have a parent and children, nesting is unlimited. The reading program receives the section number as a parameter — reading commands are placed where needed. Each section is assigned its own access level (the lowest — "guest"). In addition to CSS-formatted HTML, the text can include links to IRs with files, links to graphics, video, and audio. It is possible to generate a single HTML document with all child sections for continuous viewing.

5. Examples of projects based on the platform

• MIS "Lekar" — the most well-known medical IS. Deployed in the Bot data center: primary and backup DBs, five ASs, dedicated file server and event registration server. Database replication and dynamic load redistribution across ASs. Security certification at level 3-FL and 3-UL; a version with 3-DSP certification is available.
• AIS "Kupets" — automation of retail business processes. Modules with WWW, text, and graphical interfaces. A network of stores interacting through a central database. User connection via SaaS.
• RIAS "Kadry" — Republican Automated Information System for automating the personnel service. Operated in the Ministry of Health of the Republic of Belarus, all medical organizations of the ministry are connected. Since 2016 — by the forces of the RSCPC MT in the customer's intranet environment. Storage on PostgreSQL, programs in the WWW interface.
• ACS "Kiparis" — access control system, manufacturer Apex-Real LLC. Controllers of own production and from Russian suppliers. Storage on PostgreSQL, graphical and WWW interface, customer intranet environment.
• EQ "Kiparis" — a software and hardware complex for an electronic queue on own equipment: queue monitor, display board, TV control computer.

Options for building an IR:

• an SPC per the customer's requirements with any interface, acceptance tests, installation in the customer's environment, warranty service, refinements per partial requirements;
• an SPC per the requirements, installation in the contractor's environment, support for the entire project term, work per Agile (cloud interface only);
• connecting the customer to an existing IR of the contractor, training, work under a support contract (cloud interface only).

6. Application software development

Base languages: BCLIP (business processes) and JavaScript (browser interface). When using a browser as the interface, any languages supporting the CGI protocol are acceptable.

Text interface: the same data access options are available as for graphical applications, but instead of RDS — the SSH protocol: an order of magnitude less traffic, more economical communication channels. Used by system programmers to work with remote DB/AS. The platform allows creating text applications and using them for low-level work on remote resources. The base language is BCLIP; where no database storage is required — any Linux languages.

CGI (Common Gateway Interface): the program has no direct connection with the user; it is called by the HTTP server upon a browser request, and the response is converted into an HTTP response. HTTP traffic goes over the communication channel — the most advanced protocol in terms of security and reliability. The HTTP and HTML standards are governed by W3C — cross-platform support is guaranteed. The absence of direct communication with the operator requires mechanisms for controlling work; the platform (20+ years of experience) has developed such mechanisms in the BCLIP language for developing and maintaining complex IS. Any Linux languages are also acceptable where no database storage is required.

Graphical interface: an application on the workstation gets direct access to the remote DB over a local network/intranet/internet and sends SQL queries. An alternative is a client-server architecture with "remote procedure calls" (RPC): reduces channel requirements but complicates development (two applications instead of one). A third variant — placing the application on the AS and remote desktop (RDS) on the workstation: changes to the graphical image are transmitted over the channel (has analogues in other OSs).

7. BCLIP execution system: technical capabilities

Origin and evolution. The BCLIP execution system is a compiler compatible with the CA-Clipper (xBase) language, created at the turn of the 1990s–2000s by the Moscow-based company «АйтиК» (itk.ru) as a free project under the GPL. Its goal is to port banking and corporate applications written for CA-Clipper 5.2/5.3 under DOS to Linux/Unix without rewriting. The common ancestor of BCLIP and the Harbour project is the CA-Clipper language (Nantucket, 1985; from 1992 — Computer Associates). Harbour is a later parallel branch of the same xBase line, not an ancestor of BCLIP.

BCLIP in the BIP platform is a specialized implementation optimized for Enterprise-level Linux infrastructure and integrated with the PostgreSQL DBMS.

Language paradigm: a concise syntax for working with databases and metaprogramming elements. BCLIP is oriented toward declarative description of business logic and allows compactly expressing data operations. Key capabilities:

Database operators — a concise syntax for creating data entry screen forms with validation:

USE Customer SHARED NEW
@ 1, 0 SAY "CustNum" GET Customer->CustNum VALID Customer->CustNum > 0
READ

Macro operator (runtime compiler) — compilation and execution of an arbitrary expression at runtime, dynamic formation of database field references, code generation based on user input:

&( "Customer->" + cFieldName )
&( cExpression )

CodeBlocks (closures) — capturing variables from the external context, support for closures, delegates, and functional objects:

{ |x, y| x + y + nExternalVar }

Object-oriented programming — classes, inheritance, encapsulation, polymorphism:

CREATE CLASS Person
    VAR Name
    METHOD New( cName )
    METHOD Describe()
ENDCLASS

Database connectivity architecture. BCLIP implements the Replaceable Database Drivers (RDD) architecture, ensuring independence from a specific DBMS: PostgreSQL (primary), MySQL, SQLite, Firebird, Oracle, ODBC, as well as compatibility with xBase formats (DBF, DBFNTX, DBFCDX).

License and openness. BCLIP is distributed under the GPL (a free project by «АйтиК»). This provides vendor independence, access to the source code, and a documented syntax compatible with CA-Clipper.

Rationale for the choice. BCLIP was chosen as the execution system of the BIP platform for the following reasons:

1. Proven reliability — the technology has been in operation since 1985, with millions of applications in banking, insurance, and management.
2. High performance — compiled code with runtime optimization capability.
3. Scalability — support for multiple application servers.
4. Openness — no vendor lock-in, full code portability.
5. Import independence — open source, vendor independence.
6. Compatibility — the ability to port code from other xBase dialects (Harbour, xHarbour, FlagShip, FoxPro).

8. Database structure

Fig. 3. Platform table types and access rights separation model

The platform's database provides four types of tables (plus classifiers), differing in the method of entry, access level, and storage rules:

• T0 classifiers — master data common to all participants; service information. Only employees of the IS developer can modify it.
• T1 branch tables — information of a specific institution with the ability to query across all institutions in a single SQL query. Access — only to employees of the specific enterprise.
• T2 branch tables — information of a specific institution without the need to query across many institutions. Access — only to employees of the specific enterprise.
• T3 global tables — master data common to all. Data of one enterprise may accumulate in large volumes without affecting the system's performance for other participants.
• T4 event accounting yearbooks — common master data; tables are created for each year (month), and information is automatically distributed across tables by date. Protects the system from overflow; "old" tables can be placed on external media.

Data modification operations are reflected in the operation log — the ability to "analyze the causes" of incorrect data appearing. Responsibility for the accuracy of first-type table data is determined by the job responsibilities of the support service employees.

9. Compliance with information security requirements

Fig. 4. Protection layers and OAC certification

Various information systems deployed on the BIP platform have repeatedly passed certification for compliance with protection class requirements 3-in, 3-ul, 3-spe, and 3-DSP. Certification applies to specific deployed ISs, not to the operating system and not to the platform as such: during certification, the requirements for IS operation and the methods of connecting users are determined. If an organization connects to an already certified solution, it may not need its own certification, while a new solution built on the platform undergoes a simplified certification. As a result, a protected information space is created for processing personal, medical, and special data.

Experience with Belarusian VPN solutions. The company has experience building secure communication channels based on Belarusian VPN solutions: Bel VPN, itVPN, SCrypto. Technical support is provided for their configuration and maintenance.

Connection requirements: communication channels passing through the internet must be protected from unauthorized influence by a certified method; each local network and each workstation — via a secure channel; there is no access to the IS over an unsecured channel.

By law, users designate a responsible person — a "security officer". As a result, the enterprise processes the personal data of its clients in a protected IS and is relieved of the need to independently carry out measures to certify its local IS. The requirements of the regulatory document OAC ORDER UNDER THE PRESIDENT OF THE REPUBLIC OF BELARUS dated February 20, 2020 No. 66 "On measures to implement Decree of the President of the Republic of Belarus dated December 9, 2019 No. 449" are met.

10. Ensuring information retention and operability

Fig. 5. Parallel replication between two data centers and failover in case of primary site failure

Hardware failures of the DB (up to destruction of the disk subsystem):

• data loss — the replication subsystem guarantees, under the terms of the contract, retention of information entered more than 15 minutes ago;
• resource downtime — backup DBs restore operability within half an hour at most. This is an order of magnitude more reliable than standard daily backups.

Data center disruptions: a rarer cause, but with destructive consequences (an example — the fire at a data center in Odessa). Protection — database replication: one of the replicas in another data center. If the primary data center fails, the IS is redistributed to backup sites, reconfiguration takes about an hour. Users only need to log in from the backup address without additional actions.

Employee errors:

• an incorrectly performed operation — the "transactionality" mechanism: all changes are performed as a "package"; if an error is detected, not a single byte is saved, even if the program has already changed some records; the change package is a "transaction";
• a successfully performed operation with incorrect (but logically consistent) data — the package is recorded in the operation log and can be canceled by an authorized employee (reversal) with a return to the state "before the operation". A built-in blockchain mechanism is used, which will not allow an operation to be canceled if its results have already been used to perform the next one.

Software errors: a partially completed operation is prevented by transactionality; an incorrectly completed one is corrected by reversal through the operation log, followed by ordering a fix from the vendor.

24/7 operation: backup DBs and backup ASs are located in different data centers; for ASs — dynamic load redistribution; for DBs — redistribution of requests already at the software level depending on their types. Together, this allows scaling capacity by adding the number and power of computers.

11. Access rights separation

• Access to the IS — a hardware restrictive level: a secure communication channel; the enterprise local network must also be certified.
• Enterprise rights — a unique number and allocated data niche. Access to another enterprise's data — only within the boundaries set by the owner; part of the data is publicly available. The main criterion for participation is primacy in the data processing chain.
• Theme, role, mode — terminology: "Theme" — the name of an IR or its part (for example, "Personnel accounting"); "Role" — a subset of operations within the theme ("Subsystem setup", "Current work with personnel"); "Mode" — a set of programs within the role ("Working with the current employee card", "Hiring", etc.).

12. Organizing interaction with third-party IS

File exchange — data export/import. Supported import formats:

• TXT — a text file with an agreed format;
• JSON — a text file in JSON format with an agreed structure;
• XML — a text file with an agreed list of fields in the XML standard;
• DBF — a well-structured xBase standard format.

In addition to these formats — acceptance of files of practically any format, attached to IS objects. The same formats are also supported for transmitting data to third-party ISs, plus all HTML standard formats supported by modern browsers.

The full list of protocols and formats is given in section 3 "Communication channels" (subsection "Interaction protocols").