Why AI requires a different approach to data management and storage

AI cannot watch videos, read books or listen to podcasts

More and more companies are using AI or planning to use it. However, 95% of projects fail. The reasons for this are compliance and regulatory concerns, security risks, implementation and maintenance costs. But structural challenges such as data silos that restrict access to AI and cultural shortcomings when employees are not prepared and trained also play a role. Often, companies themselves know very little about AI.

This article explains the basics of AI-compatible data preparation and the necessary prerequisites, and presents suitable architectures and solutions. Many people now know what a large language model (LLM) is. But who knows what MLM or VLM are?

For AI, there is an important distinction. LLMs were developed for processing human language. However, images, audio and video files can contribute to relevant results.

In medicine, for example, the evaluation of images is crucial. Phishing emails can be detected more easily if images and text are analysed. If deepfakes are also used, videos must also be checked. This is where Vision Language Models (VLMs) and Multimodal Language Models (MLMs) come in.

VLMs can process visual content in addition to language; MLMs can be used to analyse audio files in addition to language and image or video content.

However, LLM, VLM and MLM have one thing in common: none of these models can do anything with native file formats or unstructured or semi-structured raw data in a data warehouse.

In the following, large, visual and multimodal language models are summarised as xLM.

The basis of intelligence: data

In order for AI to be able to use it, data must also be classified, curated, sorted, enriched, indexed and vectorised. Data preparation is critical to the success of AI projects. However, traditional data management and storage approaches are not well suited to AI and require additional measures, products and layers.

Many of the modern storage and data management platforms have been specifically designed and developed for the requirements of AI. The core of the problem lies in the way AI models process and retrieve information. xLMs cannot semantically understand the content of native file formats such as WAV, JPEG, DOCX, etc. They cannot read text, see images or hear audio. For an xLM, content must be converted into a standardised, numerically processable form before it can be used.

Formats such as DOCX or PDF also contain complex layout information (columns, headers/footers, tables, embedded images) that interrupt the pure text flow. This is confusing for a language model.

Metadata and visual elements must be removed and the pure text extracted for further processing. Formats such as JPEG, MPEG or WAV do not contain direct language or image information, but rather binary data (pixels, audio waveforms). A large language model cannot interpret this code. Images are described as text for AI using a vision model; audio or video is transcribed into text using a speech-to-text model.

The results, i.e. the text data, are then broken down into meaningful small sections (chunks or tokens) and converted into vectors, which are stored in a vector database, known as an index. This process is also referred to as embedding and is particularly important for the use of extensions such as retrieval-augmented generation (RAG).

When you ask ChatGPT a question, that question is also vectorised. The vectors of the question are compared with the index to find semantically relevant results, e.g. a section of text in a document. As new data is constantly being added or existing files are changed, the index must be continuously updated and permanently synchronised. This is critical for the quality and relevance of an AI application.

For data that changes infrequently or at fixed intervals, such as quarterly reports or archive documents, the index can be updated periodically. If data changes frequently or new data is added (e.g. support tickets, daily news feeds, e-commerce or financial transactions), the index should be updated incrementally in near real time.

The need for a vector database

Searching for similarities in a database containing millions of vectors is much faster and more efficient than extracting and analysing content from native formats or unstructured or semi-structured raw data in a vast ocean of data.

The process of preparing data for AI involves several steps. The architecture of the infrastructure and the selection of suitable components for its construction are critical to success. It is therefore important to understand the basics of this process. The individual process steps are:

During extraction, raw data is obtained from its native formats. Specialised tools such as Apache Tika or corresponding cloud services are required to extract pure text from complex structures (PDFs, DOCX, HTML, etc.) or audio/video data.

The extracted text is then processed. During cleaning, irrelevant elements (headers and footers, image captions, HTML tags) are removed and the text is normalised (e.g. correction of formatting errors).

The processed text must then be broken down into small, meaningful segments (chunks). The size is crucial. If the chunks are too large, they contain too much irrelevant information. If they are too small, the AI loses the necessary context. Overlap between the chunks ensures that no semantic connections are lost.

Each chunk is converted into a high-dimensional numerical vector using an embedding model. A vector contains the (encoded) semantic meaning of the text section.

Finally, the generated vectors are stored together with their metadata (e.g. original document ID, creation date) in a vector database (the index).

Within the AI pipeline, vectorisation is a step in the ELT/ETL process. Both ELT and ETL are data integration methods, but they differ in the order of the individual steps: extract (E), transform (T), load (L). In ETL, the data is extracted from the sources and transformed and cleaned before being loaded into the target system. In ELT, the data is loaded directly into the target system. The transformation then takes place in the target system.

Compliance vs. Performance

ETL is suitable for smaller data sets and for environments where compliance with strict regulations is important (e.g. finance or healthcare). ETL offers granular control over data processing and quality. For example, confidential information can be removed or anonymised. The disadvantage is higher latency and the need for a dedicated transformation server, which can incur additional costs.

ELT is a common method in public cloud architectures. The processing power and scalability of the target system make ELT the preferred method for large data volumes. The raw data (data in its original, unmodified state, exactly as it was extracted from the source system) can be more easily stored for later analysis and use with different schemas (schema-on-read approach) – but this also poses a risk. Raw data may contain sensitive information and violate compliance requirements if loaded in an uncontrolled manner. ELT in on-premises operation requires a powerful target system for transformation, which in turn drives up total cost of ownership.

The choice ultimately depends on the specific requirements of the project, the amount of data and compliance requirements. ELT is often used for training xLM. AI models, especially for deep learning, often require huge amounts of raw data to recognise patterns. With the ELT approach, data is quickly and completely available for different setups and models.

Since the raw data is retained, it can also be used for other business intelligence and analysis purposes. Keeping the data in its native format also preserves all the original details and nuances that can be valuable for training complex AI models. With ETL, useful information can be lost due to premature standardisation. In certain scenarios, especially when dealing with sensitive data subject to strict regulations such as the GDPR, ETL should be preferred.

The role of the data management and storage platform

Modern analysis tools and ML frameworks (such as TensorFlow, PyTorch, Spark) are designed to access data directly in distributed file systems. They have connectors and libraries to read the various native formats and perform the necessary transformations on the fly.

There are storage and database systems that perform vectorisation directly in the database or storage system. However, most manufacturers who emphasise the suitability of their storage for AI usually only mean the speed of data access.

One exception is VAST Data. While most providers of storage for AI (e.g. Quantum, NetApp or Weka) focus on the high-performance provision of data for AI workflows, VAST Data's storage systems offer an integrated unified database for structured and unstructured data. The VAST DataBase enables vectors and raw data to be stored together in the database. This allows vectors to be generated and managed for search and query without external orchestration tools or transformation servers.

The Hazelcast in-memory computing platform is an example of a dedicated transformation server or service. The Hazelcast platform combines in-memory data storage, distributed computing and vector search. Using Python clients, developers can consume unstructured data (e.g. images), calculate embeddings (using external models, but within the Hazelcast pipeline) and store them as vectors in a vector collection data structure.

Specialised vector databases such as Chroma, Weaviate, Qdrant and Milvus are designed from the ground up for efficient vector storage and search and offer integrated embedding functions. However, traditional relational or NoSQL databases can also integrate vector functions to support the RAG workflow. Examples include PostgreSQL with pgvector, Google Cloud AlloyDB, Amazon S3 Vector, SAP HANA Cloud, and Elasticsearch.

External orchestration tools such as Hammerspace or Komprise can combine data across different locations, both on-premises and in multi-clouds, into a single, global namespace. They make an important contribution to data classification and metadata management and can accelerate AI workflows. AI requires clean metadata management.

Metadata (e.g., document ID, author, creation date, last modification) is crucial for managing the index (deleting/replacing old chunks). It also helps AI filter and prioritise the relevance of search results or show a user the source of the generated response.

Classification is also crucial for more relevant conclusions (inference). Vectors are formed based on their classification. Closely related data also receive closely related vectors. Platforms such as Komprise or Ctera offer enhanced security features to protect sensitive data.

As with so many things, the choice of platforms depends on the specific requirements of the project, the amount of data to be processed and compliance requirements. However, instead of just looking for storage products in the traditional sense (file servers), it is database and data platform solutions that simplify the data pipeline process.

This article was first published in German on Computing’s sister site Computing Deutschland