Dessin de forme d’onde avec AVAssetReader

Je lis la chanson de la bibliothèque iPod en utilisant assetUrl (dans le code nommé audioUrl), je peux y jouer de nombreuses façons, je peux le couper, je peux le faire avec … mais je ne comprends vraiment pas ce que je vais faire avec ça CMSampleBufferRef pour obtenir des données pour dessiner une forme d’onde! J’ai besoin d’informations sur les valeurs maximales, comment puis-je l’obtenir (peut-être une autre)?

AVAssetTrack * songTrack = [audioUrl.tracks objectAtIndex:0]; AVAssetReaderTrackOutput * output = [[AVAssetReaderTrackOutput alloc] initWithTrack:songTrack outputSettings:nil]; [reader addOutput:output]; [output release]; NSMutableData * fullSongData = [[NSMutableData alloc] init]; [reader startReading]; while (reader.status == AVAssetReaderStatusReading){ AVAssetReaderTrackOutput * trackOutput = (AVAssetReaderTrackOutput *)[reader.outputs objectAtIndex:0]; CMSampleBufferRef sampleBufferRef = [trackOutput copyNextSampleBuffer]; if (sampleBufferRef){/* what I gonna do with this? */} 

Aidez-moi, s’il vous plaît!

Je cherchais une chose similaire et j’ai décidé de “rouler moi-même”. Je me rends compte que c’est un ancien message, mais si quelqu’un d’autre est à la recherche de cela, voici ma solution. il est relativement rapide et sale et normalise l’image à “pleine échelle”. les images qu’il crée sont «larges», c’est-à-dire que vous devez les placer dans un UIScrollView ou gérer autrement l’affichage.

ceci est basé sur des réponses données à cette question

Échantillon sortie

forme d'onde d'échantillon

EDIT: J’ai ajouté une version logarithmique des méthodes de calcul de moyenne et de rendu, voir la fin de ce message pour la version alternative et les résultats de comparaison. Personnellement, je préfère la version linéaire originale, mais j’ai décidé de l’afficher, au cas où quelqu’un pourrait améliorer l’algorithme utilisé.

Vous aurez besoin de ces importations:

 #import  #import  

Tout d’abord, une méthode de rendu générique qui prend un pointeur vers les données d’échantillon moyennées,
et renvoie un UIImage. Notez que ces échantillons ne sont pas des échantillons audio lisibles.

 -(UIImage *) audioImageGraph:(SInt16 *) samples normalizeMax:(SInt16) normalizeMax sampleCount:(NSInteger) sampleCount channelCount:(NSInteger) channelCount imageHeight:(float) imageHeight { CGSize imageSize = CGSizeMake(sampleCount, imageHeight); UIGraphicsBeginImageContext(imageSize); CGContextRef context = UIGraphicsGetCurrentContext(); CGContextSetFillColorWithColor(context, [UIColor blackColor].CGColor); CGContextSetAlpha(context,1.0); CGRect rect; rect.size = imageSize; rect.origin.x = 0; rect.origin.y = 0; CGColorRef leftcolor = [[UIColor whiteColor] CGColor]; CGColorRef rightcolor = [[UIColor redColor] CGColor]; CGContextFillRect(context, rect); CGContextSetLineWidth(context, 1.0); float halfGraphHeight = (imageHeight / 2) / (float) channelCount ; float centerLeft = halfGraphHeight; float centerRight = (halfGraphHeight*3) ; float sampleAdjustmentFactor = (imageHeight/ (float) channelCount) / (float) normalizeMax; for (NSInteger intSample = 0 ; intSample < sampleCount ; intSample ++ ) { SInt16 left = *samples++; float pixels = (float) left; pixels *= sampleAdjustmentFactor; CGContextMoveToPoint(context, intSample, centerLeft-pixels); CGContextAddLineToPoint(context, intSample, centerLeft+pixels); CGContextSetStrokeColorWithColor(context, leftcolor); CGContextStrokePath(context); if (channelCount==2) { SInt16 right = *samples++; float pixels = (float) right; pixels *= sampleAdjustmentFactor; CGContextMoveToPoint(context, intSample, centerRight - pixels); CGContextAddLineToPoint(context, intSample, centerRight + pixels); CGContextSetStrokeColorWithColor(context, rightcolor); CGContextStrokePath(context); } } // Create new image UIImage *newImage = UIGraphicsGetImageFromCurrentImageContext(); // Tidy up UIGraphicsEndImageContext(); return newImage; } 

Ensuite, une méthode qui prend un AVURLAsset et renvoie des données d'image PNG

 - (NSData *) renderPNGAudioPictogramForAsset:(AVURLAsset *)songAsset { NSError * error = nil; AVAssetReader * reader = [[AVAssetReader alloc] initWithAsset:songAsset error:&error]; AVAssetTrack * songTrack = [songAsset.tracks objectAtIndex:0]; NSDictionary* outputSettingsDict = [[NSDictionary alloc] initWithObjectsAndKeys: [NSNumber numberWithInt:kAudioFormatLinearPCM],AVFormatIDKey, // [NSNumber numberWithInt:44100.0],AVSampleRateKey, /*Not Supported*/ // [NSNumber numberWithInt: 2],AVNumberOfChannelsKey, /*Not Supported*/ [NSNumber numberWithInt:16],AVLinearPCMBitDepthKey, [NSNumber numberWithBool:NO],AVLinearPCMIsBigEndianKey, [NSNumber numberWithBool:NO],AVLinearPCMIsFloatKey, [NSNumber numberWithBool:NO],AVLinearPCMIsNonInterleaved, nil]; AVAssetReaderTrackOutput* output = [[AVAssetReaderTrackOutput alloc] initWithTrack:songTrack outputSettings:outputSettingsDict]; [reader addOutput:output]; [output release]; UInt32 sampleRate,channelCount; NSArray* formatDesc = songTrack.formatDescriptions; for(unsigned int i = 0; i < [formatDesc count]; ++i) { CMAudioFormatDescriptionRef item = (CMAudioFormatDescriptionRef)[formatDesc objectAtIndex:i]; const AudioStreamBasicDescription* fmtDesc = CMAudioFormatDescriptionGetStreamBasicDescription (item); if(fmtDesc ) { sampleRate = fmtDesc->mSampleRate; channelCount = fmtDesc->mChannelsPerFrame; // NSLog(@"channels:%u, bytes/packet: %u, sampleRate %f",fmtDesc->mChannelsPerFrame, fmtDesc->mBytesPerPacket,fmtDesc->mSampleRate); } } UInt32 bytesPerSample = 2 * channelCount; SInt16 normalizeMax = 0; NSMutableData * fullSongData = [[NSMutableData alloc] init]; [reader startReading]; UInt64 totalBytes = 0; SInt64 totalLeft = 0; SInt64 totalRight = 0; NSInteger sampleTally = 0; NSInteger samplesPerPixel = sampleRate / 50; while (reader.status == AVAssetReaderStatusReading){ AVAssetReaderTrackOutput * trackOutput = (AVAssetReaderTrackOutput *)[reader.outputs objectAtIndex:0]; CMSampleBufferRef sampleBufferRef = [trackOutput copyNextSampleBuffer]; if (sampleBufferRef){ CMBlockBufferRef blockBufferRef = CMSampleBufferGetDataBuffer(sampleBufferRef); size_t length = CMBlockBufferGetDataLength(blockBufferRef); totalBytes += length; NSAutoreleasePool *wader = [[NSAutoreleasePool alloc] init]; NSMutableData * data = [NSMutableData dataWithLength:length]; CMBlockBufferCopyDataBytes(blockBufferRef, 0, length, data.mutableBytes); SInt16 * samples = (SInt16 *) data.mutableBytes; int sampleCount = length / bytesPerSample; for (int i = 0; i < sampleCount ; i ++) { SInt16 left = *samples++; totalLeft += left; SInt16 right; if (channelCount==2) { right = *samples++; totalRight += right; } sampleTally++; if (sampleTally > samplesPerPixel) { left = totalLeft / sampleTally; SInt16 fix = abs(left); if (fix > normalizeMax) { normalizeMax = fix; } [fullSongData appendBytes:&left length:sizeof(left)]; if (channelCount==2) { right = totalRight / sampleTally; SInt16 fix = abs(right); if (fix > normalizeMax) { normalizeMax = fix; } [fullSongData appendBytes:&right length:sizeof(right)]; } totalLeft = 0; totalRight = 0; sampleTally = 0; } } [wader drain]; CMSampleBufferInvalidate(sampleBufferRef); CFRelease(sampleBufferRef); } } NSData * finalData = nil; if (reader.status == AVAssetReaderStatusFailed || reader.status == AVAssetReaderStatusUnknown){ // Something went wrong. return nil return nil; } if (reader.status == AVAssetReaderStatusCompleted){ NSLog(@"rendering output graphics using normalizeMax %d",normalizeMax); UIImage *test = [self audioImageGraph:(SInt16 *) fullSongData.bytes normalizeMax:normalizeMax sampleCount:fullSongData.length / 4 channelCount:2 imageHeight:100]; finalData = imageToData(test); } [fullSongData release]; [reader release]; return finalData; } 

Option avancée: Enfin, si vous souhaitez pouvoir lire l’audio à l’aide d’AVAudioPlayer, vous devrez le mettre en cache dans le dossier cache de l’ensemble des applications. Comme je le faisais, j'ai décidé de mettre également en cache les données d'image et de les regrouper dans une catégorie UIImage. vous devez inclure cette offre open source pour extraire l'audio, et du code d' ici pour gérer certaines fonctionnalités de threading d'arrière-plan.

en premier lieu, quelques définitions, et quelques méthodes de classe génériques pour gérer les noms de chemins, etc.

 //#define imgExt @"jpg" //#define imageToData(x) UIImageJPEGRepresentation(x,4) #define imgExt @"png" #define imageToData(x) UIImagePNGRepresentation(x) + (NSSsortingng *) assetCacheFolder { NSArray *assetFolderRoot = NSSearchPathForDirectoriesInDomains(NSCachesDirectory, NSUserDomainMask, YES); return [NSSsortingng ssortingngWithFormat:@"%@/audio", [assetFolderRoot objectAtIndex:0]]; } + (NSSsortingng *) cachedAudioPictogramPathForMPMediaItem:(MPMediaItem*) item { NSSsortingng *assetFolder = [[self class] assetCacheFolder]; NSNumber * libraryId = [item valueForProperty:MPMediaItemPropertyPersistentID]; NSSsortingng *assetPictogramFilename = [NSSsortingng ssortingngWithFormat:@"asset_%@.%@",libraryId,imgExt]; return [NSSsortingng ssortingngWithFormat:@"%@/%@", assetFolder, assetPictogramFilename]; } + (NSSsortingng *) cachedAudioFilepathForMPMediaItem:(MPMediaItem*) item { NSSsortingng *assetFolder = [[self class] assetCacheFolder]; NSURL * assetURL = [item valueForProperty:MPMediaItemPropertyAssetURL]; NSNumber * libraryId = [item valueForProperty:MPMediaItemPropertyPersistentID]; NSSsortingng *assetFileExt = [[[assetURL path] lastPathComponent] pathExtension]; NSSsortingng *assetFilename = [NSSsortingng ssortingngWithFormat:@"asset_%@.%@",libraryId,assetFileExt]; return [NSSsortingng ssortingngWithFormat:@"%@/%@", assetFolder, assetFilename]; } + (NSURL *) cachedAudioURLForMPMediaItem:(MPMediaItem*) item { NSSsortingng *assetFilepath = [[self class] cachedAudioFilepathForMPMediaItem:item]; return [NSURL fileURLWithPath:assetFilepath]; } 

Maintenant, la méthode init qui fait "le business"

 - (id) initWithMPMediaItem:(MPMediaItem*) item completionBlock:(void (^)(UIImage* delayedImagePreparation))completionBlock { NSFileManager *fman = [NSFileManager defaultManager]; NSSsortingng *assetPictogramFilepath = [[self class] cachedAudioPictogramPathForMPMediaItem:item]; if ([fman fileExistsAtPath:assetPictogramFilepath]) { NSLog(@"Returning cached waveform pictogram: %@",[assetPictogramFilepath lastPathComponent]); self = [self initWithContentsOfFile:assetPictogramFilepath]; return self; } NSSsortingng *assetFilepath = [[self class] cachedAudioFilepathForMPMediaItem:item]; NSURL *assetFileURL = [NSURL fileURLWithPath:assetFilepath]; if ([fman fileExistsAtPath:assetFilepath]) { NSLog(@"scanning cached audio data to create UIImage file: %@",[assetFilepath lastPathComponent]); [assetFileURL retain]; [assetPictogramFilepath retain]; [NSThread MCSM_performBlockInBackground: ^{ AVURLAsset *asset = [[AVURLAsset alloc] initWithURL:assetFileURL options:nil]; NSData *waveFormData = [self renderPNGAudioPictogramForAsset:asset]; [waveFormData writeToFile:assetPictogramFilepath atomically:YES]; [assetFileURL release]; [assetPictogramFilepath release]; if (completionBlock) { [waveFormData retain]; [NSThread MCSM_performBlockOnMainThread:^{ UIImage *result = [UIImage imageWithData:waveFormData]; NSLog(@"returning rendered pictogram on main thread (%d bytes %@ data in UIImage %0.0fx %0.0f pixels)",waveFormData.length,[imgExt uppercaseSsortingng],result.size.width,result.size.height); completionBlock(result); [waveFormData release]; }]; } }]; return nil; } else { NSSsortingng *assetFolder = [[self class] assetCacheFolder]; [fman createDirectoryAtPath:assetFolder withIntermediateDirectories:YES atsortingbutes:nil error:nil]; NSLog(@"Preparing to import audio asset data %@",[assetFilepath lastPathComponent]); [assetPictogramFilepath retain]; [assetFileURL retain]; TSLibraryImport* import = [[TSLibraryImport alloc] init]; NSURL * assetURL = [item valueForProperty:MPMediaItemPropertyAssetURL]; [import importAsset:assetURL toURL:assetFileURL completionBlock:^(TSLibraryImport* import) { //check the status and error properties of //TSLibraryImport if (import.error) { NSLog (@"audio data import failed:%@",import.error); } else{ NSLog (@"Creating waveform pictogram file: %@", [assetPictogramFilepath lastPathComponent]); AVURLAsset *asset = [[AVURLAsset alloc] initWithURL:assetFileURL options:nil]; NSData *waveFormData = [self renderPNGAudioPictogramForAsset:asset]; [waveFormData writeToFile:assetPictogramFilepath atomically:YES]; if (completionBlock) { [waveFormData retain]; [NSThread MCSM_performBlockOnMainThread:^{ UIImage *result = [UIImage imageWithData:waveFormData]; NSLog(@"returning rendered pictogram on main thread (%d bytes %@ data in UIImage %0.0fx %0.0f pixels)",waveFormData.length,[imgExt uppercaseSsortingng],result.size.width,result.size.height); completionBlock(result); [waveFormData release]; }]; } } [assetPictogramFilepath release]; [assetFileURL release]; } ]; return nil; } } 

Un exemple d'invocation de ceci:

 -(void) importMediaItem { MPMediaItem* item = [self mediaItem]; // since we will be needing this for playback, save the url to the cached audio. [url release]; url = [[UIImage cachedAudioURLForMPMediaItem:item] retain]; [waveFormImage release]; waveFormImage = [[UIImage alloc ] initWithMPMediaItem:item completionBlock:^(UIImage* delayedImagePreparation){ waveFormImage = [delayedImagePreparation retain]; [self displayWaveFormImage]; }]; if (waveFormImage) { [waveFormImage retain]; [self displayWaveFormImage]; } } 

Version logarithmique des méthodes de moyenne et de rendu

 #define absX(x) (x<0?0-x:x) #define minMaxX(x,mn,mx) (x< =mn?mn:(x>=mx?mx:x)) #define noiseFloor (-90.0) #define decibel(amplitude) (20.0 * log10(absX(amplitude)/32767.0)) -(UIImage *) audioImageLogGraph:(Float32 *) samples normalizeMax:(Float32) normalizeMax sampleCount:(NSInteger) sampleCount channelCount:(NSInteger) channelCount imageHeight:(float) imageHeight { CGSize imageSize = CGSizeMake(sampleCount, imageHeight); UIGraphicsBeginImageContext(imageSize); CGContextRef context = UIGraphicsGetCurrentContext(); CGContextSetFillColorWithColor(context, [UIColor blackColor].CGColor); CGContextSetAlpha(context,1.0); CGRect rect; rect.size = imageSize; rect.origin.x = 0; rect.origin.y = 0; CGColorRef leftcolor = [[UIColor whiteColor] CGColor]; CGColorRef rightcolor = [[UIColor redColor] CGColor]; CGContextFillRect(context, rect); CGContextSetLineWidth(context, 1.0); float halfGraphHeight = (imageHeight / 2) / (float) channelCount ; float centerLeft = halfGraphHeight; float centerRight = (halfGraphHeight*3) ; float sampleAdjustmentFactor = (imageHeight/ (float) channelCount) / (normalizeMax - noiseFloor) / 2; for (NSInteger intSample = 0 ; intSample < sampleCount ; intSample ++ ) { Float32 left = *samples++; float pixels = (left - noiseFloor) * sampleAdjustmentFactor; CGContextMoveToPoint(context, intSample, centerLeft-pixels); CGContextAddLineToPoint(context, intSample, centerLeft+pixels); CGContextSetStrokeColorWithColor(context, leftcolor); CGContextStrokePath(context); if (channelCount==2) { Float32 right = *samples++; float pixels = (right - noiseFloor) * sampleAdjustmentFactor; CGContextMoveToPoint(context, intSample, centerRight - pixels); CGContextAddLineToPoint(context, intSample, centerRight + pixels); CGContextSetStrokeColorWithColor(context, rightcolor); CGContextStrokePath(context); } } // Create new image UIImage *newImage = UIGraphicsGetImageFromCurrentImageContext(); // Tidy up UIGraphicsEndImageContext(); return newImage; } - (NSData *) renderPNGAudioPictogramLogForAsset:(AVURLAsset *)songAsset { NSError * error = nil; AVAssetReader * reader = [[AVAssetReader alloc] initWithAsset:songAsset error:&error]; AVAssetTrack * songTrack = [songAsset.tracks objectAtIndex:0]; NSDictionary* outputSettingsDict = [[NSDictionary alloc] initWithObjectsAndKeys: [NSNumber numberWithInt:kAudioFormatLinearPCM],AVFormatIDKey, // [NSNumber numberWithInt:44100.0],AVSampleRateKey, /*Not Supported*/ // [NSNumber numberWithInt: 2],AVNumberOfChannelsKey, /*Not Supported*/ [NSNumber numberWithInt:16],AVLinearPCMBitDepthKey, [NSNumber numberWithBool:NO],AVLinearPCMIsBigEndianKey, [NSNumber numberWithBool:NO],AVLinearPCMIsFloatKey, [NSNumber numberWithBool:NO],AVLinearPCMIsNonInterleaved, nil]; AVAssetReaderTrackOutput* output = [[AVAssetReaderTrackOutput alloc] initWithTrack:songTrack outputSettings:outputSettingsDict]; [reader addOutput:output]; [output release]; UInt32 sampleRate,channelCount; NSArray* formatDesc = songTrack.formatDescriptions; for(unsigned int i = 0; i < [formatDesc count]; ++i) { CMAudioFormatDescriptionRef item = (CMAudioFormatDescriptionRef)[formatDesc objectAtIndex:i]; const AudioStreamBasicDescription* fmtDesc = CMAudioFormatDescriptionGetStreamBasicDescription (item); if(fmtDesc ) { sampleRate = fmtDesc->mSampleRate; channelCount = fmtDesc->mChannelsPerFrame; // NSLog(@"channels:%u, bytes/packet: %u, sampleRate %f",fmtDesc->mChannelsPerFrame, fmtDesc->mBytesPerPacket,fmtDesc->mSampleRate); } } UInt32 bytesPerSample = 2 * channelCount; Float32 normalizeMax = noiseFloor; NSLog(@"normalizeMax = %f",normalizeMax); NSMutableData * fullSongData = [[NSMutableData alloc] init]; [reader startReading]; UInt64 totalBytes = 0; Float64 totalLeft = 0; Float64 totalRight = 0; Float32 sampleTally = 0; NSInteger samplesPerPixel = sampleRate / 50; while (reader.status == AVAssetReaderStatusReading){ AVAssetReaderTrackOutput * trackOutput = (AVAssetReaderTrackOutput *)[reader.outputs objectAtIndex:0]; CMSampleBufferRef sampleBufferRef = [trackOutput copyNextSampleBuffer]; if (sampleBufferRef){ CMBlockBufferRef blockBufferRef = CMSampleBufferGetDataBuffer(sampleBufferRef); size_t length = CMBlockBufferGetDataLength(blockBufferRef); totalBytes += length; NSAutoreleasePool *wader = [[NSAutoreleasePool alloc] init]; NSMutableData * data = [NSMutableData dataWithLength:length]; CMBlockBufferCopyDataBytes(blockBufferRef, 0, length, data.mutableBytes); SInt16 * samples = (SInt16 *) data.mutableBytes; int sampleCount = length / bytesPerSample; for (int i = 0; i < sampleCount ; i ++) { Float32 left = (Float32) *samples++; left = decibel(left); left = minMaxX(left,noiseFloor,0); totalLeft += left; Float32 right; if (channelCount==2) { right = (Float32) *samples++; right = decibel(right); right = minMaxX(right,noiseFloor,0); totalRight += right; } sampleTally++; if (sampleTally > samplesPerPixel) { left = totalLeft / sampleTally; if (left > normalizeMax) { normalizeMax = left; } // NSLog(@"left average = %f, normalizeMax = %f",left,normalizeMax); [fullSongData appendBytes:&left length:sizeof(left)]; if (channelCount==2) { right = totalRight / sampleTally; if (right > normalizeMax) { normalizeMax = right; } [fullSongData appendBytes:&right length:sizeof(right)]; } totalLeft = 0; totalRight = 0; sampleTally = 0; } } [wader drain]; CMSampleBufferInvalidate(sampleBufferRef); CFRelease(sampleBufferRef); } } NSData * finalData = nil; if (reader.status == AVAssetReaderStatusFailed || reader.status == AVAssetReaderStatusUnknown){ // Something went wrong. Handle it. } if (reader.status == AVAssetReaderStatusCompleted){ // You're done. It worked. NSLog(@"rendering output graphics using normalizeMax %f",normalizeMax); UIImage *test = [self audioImageLogGraph:(Float32 *) fullSongData.bytes normalizeMax:normalizeMax sampleCount:fullSongData.length / (sizeof(Float32) * 2) channelCount:2 imageHeight:100]; finalData = imageToData(test); } [fullSongData release]; [reader release]; return finalData; } 

résultats de comparaison

Linéaire
Tracé linéaire pour le démarrage de "Warm It Up" par Acme Swing Company

logarithmique
Tracé logarithmique pour le début de "Warm It Up" par Acme Swing Company

Vous devriez pouvoir obtenir un tampon audio à partir de votre sampleBuffRef, puis parcourir ces valeurs pour créer votre forme d’onde:

 CMBlockBufferRef buffer = CMSampleBufferGetDataBuffer( sampleBufferRef ); CMItemCount numSamplesInBuffer = CMSampleBufferGetNumSamples(sampleBufferRef); AudioBufferList audioBufferList; CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer( sampleBufferRef, NULL, &audioBufferList, sizeof(audioBufferList), NULL, NULL, kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment, &buffer ); // this copies your audio out to a temp buffer but you should be able to iterate through this buffer instead SInt32* readBuffer = (SInt32 *)malloc(numSamplesInBuffer * sizeof(SInt32)); memcpy( readBuffer, audioBufferList.mBuffers[0].mData, numSamplesInBuffer*sizeof(SInt32));